Oct 30 19:24:04.999839 ip-10-0-134-200 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:24:05.457227 ip-10-0-134-200 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:05.457227 ip-10-0-134-200 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:24:05.457227 ip-10-0-134-200 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:05.457227 ip-10-0-134-200 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:05.462603 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462504    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.462603 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462602    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.462603 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462606    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462609    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462612    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462615    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462619    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462622    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462625    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462628    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462630    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462633    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462636    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462639    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462641    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462644    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462652    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462655    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462658    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462663    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462667    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.462699 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462670    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462673    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462675    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462679    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462682    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462685    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462688    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462690    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462693    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462695    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462698    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462701    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462705    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462708    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462711    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462713    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462716    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462719    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462722    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.463161 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462726    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462730    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462733    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462736    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462739    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462742    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462745    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462748    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462751    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462754    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462757    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462759    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462762    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462764    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462767    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462770    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462773    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462776    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462778    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462781    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.463629 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462783    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462786    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462788    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462791    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462794    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462796    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462799    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462801    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462804    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462806    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462809    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462811    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462814    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462816    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462819    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462821    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462823    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462826    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462828    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462831    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.464103 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462833    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462836    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462839    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462841    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462844    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462846    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462850    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462853    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462856    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462858    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462861    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462863    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462866    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462868    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462871    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462873    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462876    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462878    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462881    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462884    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.464599 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462886    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462888    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462891    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462893    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462896    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.462898    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463725    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463733    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463736    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463739    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463742    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463744    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463747    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463750    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463753    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463756    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463759    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463762    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463765    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463768    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.465093 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463770    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463773    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463776    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463778    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463781    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463784    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463786    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463789    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463791    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463794    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463796    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463799    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463801    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463804    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463807    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463809    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463812    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463814    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463817    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.465630 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463819    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463822    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463825    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463827    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463830    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463832    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463834    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463837    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463840    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463843    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463846    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463849    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463852    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463854    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463857    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463861    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463864    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463867    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463870    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.466086 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463873    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463876    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463878    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463881    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463883    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463886    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463888    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463892    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463895    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463897    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463900    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463902    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463905    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463908    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463910    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463913    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463916    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463918    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463921    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.466532 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463923    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463926    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463929    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463932    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463934    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463939    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463944    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463946    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463949    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463952    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463954    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463957    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463959    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463962    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463964    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463966    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463969    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463972    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463974    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463976    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.467004 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463979    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463982    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463985    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463987    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463990    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463992    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463994    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463997    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.463999    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464002    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464005    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464008    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464011    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464013    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464015    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464579    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464588    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464592    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464596    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464600    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.467488 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464603    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464605    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464608    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464611    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464614    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464617    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464619    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464622    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464625    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464628    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464630    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464633    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464636    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464638    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464641    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464643    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464646    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464648    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464651    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464654    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.467958 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464656    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464659    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464665    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464668    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464670    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464673    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464675    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464678    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464681    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464684    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464687    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464690    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464692    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464695    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464697    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464700    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464702    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464705    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464709    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464713    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.468428 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464716    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464720    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464722    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464725    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464728    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464731    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464733    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464736    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464738    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464741    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464743    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464746    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464749    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464751    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464755    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464757    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464760    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464762    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464765    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.468910 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464768    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464770    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464773    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464777    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464782    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464785    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464788    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464791    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464794    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464797    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464799    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464802    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464804    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464807    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464809    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464812    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464814    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464817    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464819    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.469369 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464822    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464824    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464827    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464829    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464832    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464834    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464837    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464840    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464844    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464847    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464850    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464852    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464855    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464858    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464861    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464864    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464866    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464869    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464872    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.469913 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464874    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464877    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464879    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.464882    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465854    2580 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465864    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465872    2580 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465877    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465881    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465885    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465890    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465894    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465897    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465901    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465905    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465908    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465911    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465914    2580 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465918    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465921    2580 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465924    2580 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465927    2580 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465932    2580 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465935    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:24:05.470373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465938    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465941    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465945    2580 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465949    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465952    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465955    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465959    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465962    2580 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465965    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465968    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465972    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465975    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465979    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465982    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465985    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465989    2580 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465992    2580 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.465995    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466000    2580 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466003    2580 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466006    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466009    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466013    2580 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466018    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:24:05.471580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466021    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466024    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466028    2580 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466031    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466034    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466038    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466041    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466044    2580 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466047    2580 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466050    2580 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466054    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466058    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466061    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466064    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466067    2580 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466071    2580 flags.go:64] FLAG: --help="false"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466074    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466077    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466081    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466083    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466087    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466091    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466094    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:24:05.472175 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466097    2580 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466100    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466103    2580 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466106    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466109    2580 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466112    2580 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466115    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466118    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466121    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466124    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466127    2580 flags.go:64] FLAG: --lock-file=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466130    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466133    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466136    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466141    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466144    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466147    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466150    2580 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466153    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466157    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466160    2580 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466163    2580 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466167    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466170    2580 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466174    2580 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:24:05.472742 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466177    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466181    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466183    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466187    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466190    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466192    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466195    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466202    2580 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466205    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466208    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466211    2580 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466214    2580 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466217    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466220    2580 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466223    2580 flags.go:64] FLAG: --port="10250"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466226    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466229    2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e02bbff0b14edfa8"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466250    2580 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466254    2580 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466258    2580 flags.go:64] FLAG: --register-node="true"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466261    2580 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466267    2580 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466271    2580 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466273    2580 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466276    2580 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466280    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:24:05.473322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466283    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466286    2580 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466289    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466292    2580 flags.go:64] FLAG: --runonce="false"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466295    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466298    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466301    2580 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466305    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466308    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466311    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466315    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466318    2580 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466321    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466324    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466326    2580 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466329    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466332    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466335    2580 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466338    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466344    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466347    2580 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466350    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466354    2580 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466358    2580 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466361    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:24:05.473959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466364    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466369    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466372    2580 flags.go:64] FLAG: --v="2"
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466376    2580 flags.go:64] FLAG: --version="false"
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466381    2580 flags.go:64] FLAG: --vmodule=""
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466385    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466388    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466526    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466530    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466534    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466537    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466540    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466543    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466545    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466548    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466551    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466554    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466557    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466572    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466575    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466577    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.474547 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466580    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466583    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466585    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466588    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466591    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466593    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466596    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466599    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466601    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466605    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466608    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466611    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466615    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466618    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466620    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466623    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466625    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466628    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466630    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466633    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.475099 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466635    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466638    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466641    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466644    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466646    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466649    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466651    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466654    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466657    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466661    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466664    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466667    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466671    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466675    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466678    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466681    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466684    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466686    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.475594 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466689    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466692    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466695    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466699    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466702    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466704    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466708    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466711    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466713    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466716    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466718    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466721    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466724    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466726    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466729    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466731    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466734    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466736    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466739    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.476026 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466741    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466744    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466747    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466749    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466752    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466755    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466758    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466760    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466763    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466765    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466768    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466771    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466773    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466776    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466778    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466781    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466784    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466787    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466790    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466793    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.476526 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466796    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466799    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466802    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466804    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466807    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466809    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466812    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466815    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466817    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466820    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466823    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466825    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466827    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466830    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466833    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.466858    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:05.477259 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466895    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466899    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466902    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466905    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466907    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466910    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466913    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466915    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466918    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466920    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466923    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466925    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466928    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466932    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466934    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466938    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466941    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466943    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466946    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466948    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.477765 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466951    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466953    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466956    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466958    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466961    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466963    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466966    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466968    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466971    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466974    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466976    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466979    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466981    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466984    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466987    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466989    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466992    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466994    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466997    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.466999    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.478247 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467002    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467005    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467008    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467010    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467013    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467017    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467020    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467024    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467026    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467029    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467033    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467037    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467040    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467044    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467048    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467052    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467055    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467059    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.478934 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467062    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467065    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467068    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467070    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467073    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467075    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467078    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467080    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467084    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467086    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467089    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467091    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467094    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467096    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467099    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467101    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467104    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467106    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467109    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.479462 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467113    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467116    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467121    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467123    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467126    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467129    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467131    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467134    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467136    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467139    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467142    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467144    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467147    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467149    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467152    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467155    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467157    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467160    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467162    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467165    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.479936 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467167    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467170    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467173    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467176    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467178    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467181    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467183    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467187    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.467189    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.475738    2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.475757    2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.475773    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.475779    2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475842    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475847    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475851    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475854    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475857    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475874    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.480398 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475879    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475882    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475885    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475888    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475891    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475894    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475896    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475899    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475902    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475904    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475907    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475909    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475912    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475915    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475917    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475920    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475922    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475925    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475928    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.480900 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475930    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475932    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475935    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475938    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475947    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475950    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475953    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475956    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475958    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475961    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475963    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475966    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475968    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475971    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475973    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475976    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475978    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475981    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475983    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.481340 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475986    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475988    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475991    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475993    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475996    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.475998    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476001    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476003    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476007    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476011    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476014    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476017    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476019    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476022    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476024    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476027    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476030    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476033    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476041    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476044    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.481794 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476047    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476049    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476052    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476055    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476057    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476060    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476063    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476065    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476067    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476070    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476073    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476075    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476078    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476080    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476082    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476085    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476088    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476091    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476095    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476098    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.482317 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476100    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476103    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476106    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476108    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476111    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476113    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476116    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476118    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476121    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476124    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476127    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476134    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476137    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476140    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476143    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476146    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476148    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476151    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476153    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.482798 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476156    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476158    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.476161    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.476189    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479005    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479019    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479023    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479026    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479029    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479033    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479036    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479038    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479041    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479044    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479047    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479049    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.483355 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479052    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479054    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479057    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479060    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479063    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479066    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479069    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479072    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479074    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479077    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479080    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479082    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479085    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479088    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479091    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479093    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479096    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479098    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479102    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.483882 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479104    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479107    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479109    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479112    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479117    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479122    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479125    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479128    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479130    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479133    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479136    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479138    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479141    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479143    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479146    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479148    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479151    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479154    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479156    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.484366 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479159    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479164    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479167    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479170    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479173    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479177    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479179    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479182    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479185    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479187    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479190    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479193    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479195    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479198    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479200    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479203    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479206    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479208    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479211    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479213    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.484873 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479216    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479218    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479220    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479223    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479225    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479228    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479231    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479233    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479236    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479239    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479241    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479244    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479246    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479256    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479259    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479261    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479265    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479269    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479272    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.485678 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479274    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479277    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479279    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479282    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479285    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479288    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479290    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479293    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479295    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479298    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479300    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479303    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479305    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479308    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479310    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479313    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.486327 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479315    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.479344    2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479394    2580 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479399    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479402    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479405    2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479408    2580 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479411    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479414    2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479417    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479429    2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479438    2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479441    2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479444    2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479447    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479449    2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:05.486730 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479452    2580 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479455    2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479457    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479460    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479463    2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479466    2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479468    2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479471    2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479474    2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479477    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479479    2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479482    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479485    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479487    2580 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479490    2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479492    2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479495    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479497    2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479500    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:05.487182 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479502    2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479505    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479508    2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479510    2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479513    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479517    2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479521    2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479524    2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479527    2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479531    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479534    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479536    2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479539    2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479541    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479544    2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479546    2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479549    2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479551    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479554    2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:05.487825 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479556    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479572    2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479576    2580 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479579    2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479581    2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479585    2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479588    2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479591    2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479593    2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479596    2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479599    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479602    2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479605    2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479608    2580 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479610    2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479613    2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479615    2580 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479618    2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479621    2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479623    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:05.488313 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479625    2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479628    2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479631    2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479634    2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479636    2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479639    2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479641    2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479644    2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479646    2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479649    2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479652    2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479654    2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479657    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479659    2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479662    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479664    2580 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479667    2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479672    2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479676    2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:05.488896 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479679    2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479682    2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479688    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479691    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479693    2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479696    2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479699    2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479701    2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479704    2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479707    2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479709    2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479712    2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479714    2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479717    2580 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:05.479719    2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.479784    2580 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.481975    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.482936    2580 server.go:1014] "Starting client certificate rotation"
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.483039    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:24:05.489343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.483090    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:24:05.489831 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.485076    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:24:05.489831 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.486245    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:24:05.503946 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.503931    2580 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:24:05.509128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.509109    2580 log.go:25] "Validated CRI v1 image API"
Oct 30 19:24:05.509996 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.509974    2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:24:05.513256 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.513237    2580 fs.go:135] Filesystem UUIDs: map[2ca204d7-d8fe-495c-a6fa-ac988d898c9a:/dev/nvme0n1p3 7695ae5c-5d6c-44a0-a762-5b94a71d3819:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:24:05.513308 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.513257    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:24:05.518750 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.518645    2580 manager.go:217] Machine: {Timestamp:2026-10-30 19:24:05.518170013 +0000 UTC m=+0.391052798 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099781 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255c5ea6af2d35681076f152e1ca55 SystemUUID:ec255c5e-a6af-2d35-6810-76f152e1ca55 BootID:0983f3f5-d04b-419c-a8f8-301b85a8c190 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c7:76:0d:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c7:76:0d:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:ad:73:5d:d2:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:24:05.519380 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.519369    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:24:05.519472 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.519460    2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:24:05.523268 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.523234    2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:24:05.523437 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.523271    2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-200.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:24:05.523519 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.523442    2580 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:24:05.523519 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.523455    2580 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:24:05.523519 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.523469    2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:24:05.524325 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.524313    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:24:05.525280 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.525266    2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:24:05.528855 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.528840    2580 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:24:05.528997 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.528986    2580 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:24:05.529815 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.529804    2580 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:24:05.529872 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.529822    2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:24:05.529872 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.529838    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:24:05.529872 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.529851    2580 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:24:05.529872 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.529863    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:24:05.530193 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.530180    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:24:05.533057 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.533036    2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:24:05.539508 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.539492    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:24:05.540749 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540691    2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:24:05.540802 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540779    2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:24:05.540962 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540952    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:24:05.540991 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540968    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:24:05.540991 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540975    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:24:05.540991 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540982    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:24:05.540991 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540989    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:24:05.541099 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.540995    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:24:05.541992 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.541981    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:24:05.542026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.541993    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:24:05.542026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542002    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:24:05.542026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542009    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:24:05.542026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542018    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:24:05.542026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542027    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:24:05.542975 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542965    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:24:05.543010 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.542979    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:24:05.547085 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.547073    2580 server.go:1263] "Started kubelet"
Oct 30 19:24:05.547227 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.547173    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:24:05.547293 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.547193    2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:24:05.547293 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.547251    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:24:05.547730 ip-10-0-134-200 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:24:05.548495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.548479    2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:24:05.551334 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.551317    2580 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:24:05.554821 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.554804    2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:24:05.556486 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.556456    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:24:05.557534 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.557513    2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:24:05.557651 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.557616    2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:05.558236 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558212    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:24:05.558346 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558215    2580 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:24:05.558346 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558267    2580 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:24:05.558461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558445    2580 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:24:05.558461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558461    2580 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:24:05.558629 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558544    2580 factory.go:55] Registering systemd factory
Oct 30 19:24:05.558629 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558575    2580 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:24:05.558983 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558808    2580 factory.go:153] Registering CRI-O factory
Oct 30 19:24:05.558983 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558823    2580 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:24:05.558983 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558862    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:24:05.558983 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558891    2580 factory.go:103] Registering Raw factory
Oct 30 19:24:05.558983 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.558901    2580 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:24:05.560219 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.559990    2580 manager.go:319] Starting recovery of all containers
Oct 30 19:24:05.572031 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.571904    2580 manager.go:324] Recovery completed
Oct 30 19:24:05.575798 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.575783    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.577540 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.577523    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.577621 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.577549    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.577621 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.577575    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.578064 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578052    2580 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:24:05.578064 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578063    2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:24:05.578143 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578078    2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:24:05.578143 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578054    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.578824 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578809    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.578879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578836    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.578879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.578848    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.579439 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.579428    2580 policy_none.go:50] "Start"
Oct 30 19:24:05.579471 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.579443    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:24:05.579471 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.579453    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:24:05.580682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.580672    2580 policy_none.go:44] "Start"
Oct 30 19:24:05.610329 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.610298    2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:24:05.613537 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613455    2580 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:24:05.613537 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.613503    2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:24:05.613537 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613519    2580 server.go:87] "Starting device plugin registration server"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613805    2580 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613820    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613921    2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.613993    2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.614001    2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.615723    2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.615764    2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.628925    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.630043    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.630064    2580 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.630080    2580 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:24:05.631541 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.630113    2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:24:05.684788 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.684752    2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-200.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:24:05.684788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.684769    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-200.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:24:05.714947 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.714870    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.717408 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.717391    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.717496 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.717423    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.717496 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.717437    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.717496 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.717463    2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.730200 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.730179    2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"]
Oct 30 19:24:05.730276 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.730234    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.731128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.731111    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.731221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.731142    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.731221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.731155    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.731788 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.730624    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-200.us-west-2.compute.internal.18e3651b88309c55  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-200.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-200.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 19:24:05.546998869 +0000 UTC m=+0.419881652,LastTimestamp:2026-10-30 19:24:05.546998869 +0000 UTC m=+0.419881652,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-200.us-west-2.compute.internal,}"
Oct 30 19:24:05.732499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.732487    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.732659 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.732643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.732691 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.732673    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.733205 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733187    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.733300 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733213    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.733300 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733227    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.733300 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733245    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.733300 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733261    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.733300 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.733275    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.734489 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.734473    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.734582 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.734500    2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:05.735369 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.735350    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:05.735451 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.735380    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:05.735451 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.735396    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:05.748075 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.747831    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 19:24:05.748164 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.747831    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 19:24:05.748164 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.747933    2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 19:24:05.754817 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.754795    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:24:05.754913 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.754847    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ll6qm"
Oct 30 19:24:05.760070 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.760050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.760167 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.760083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.760167 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.760107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.764699 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.764680    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-200.us-west-2.compute.internal\" not found" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.767960 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.767940    2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-200.us-west-2.compute.internal\" not found" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.838126 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.838107    2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:05.843730 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.843713    2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.843802 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:05.843733    2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-200.us-west-2.compute.internal\": node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:05.860648 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.860713 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860652    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.860713 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.860713 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860693    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.860713 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860699    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.860830 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.860699    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:05.958438 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:05.958409    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:06.065781 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.065717    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:06.068395 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.068380    2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:06.068525 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.068509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:06.072481 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.072465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:06.125885 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.125866    2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:06.483734 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.483710    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:24:06.484406 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.483864    2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a0c1794a904604bf0bf421b952c83549-4b7c0d71fd7f1dce.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-200.us-west-2.compute.internal.18e3651b8a02973f\": read tcp 10.0.134.200:54704->18.227.202.105:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-200.us-west-2.compute.internal.18e3651b8a02973f  default   4798 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-200.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-200.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-200.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 19:24:05 +0000 UTC,LastTimestamp:2026-10-30 19:24:05.731126834 +0000 UTC m=+0.604009618,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-200.us-west-2.compute.internal,}"
Oct 30 19:24:06.533374 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.533342    2580 apiserver.go:52] "Watching apiserver"
Oct 30 19:24:06.558072 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.558040    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:24:06.702798 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.702762    2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:06.759051 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.758947    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:19:06 +0000 UTC" deadline="2028-07-14 08:38:00.194055812 +0000 UTC"
Oct 30 19:24:06.759051 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.758991    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14941h13m53.435068326s"
Oct 30 19:24:06.802741 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.802703    2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:24:06.803034 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.803016    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lf6jh","openshift-network-operator/iptables-alerter-xs627","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-cjw9x","openshift-ovn-kubernetes/ovnkube-node-rx5vx","kube-system/global-pull-secret-syncer-vg85t","openshift-multus/multus-additional-cni-plugins-q6ssb","kube-system/konnectivity-agent-qhxdt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7","openshift-image-registry/node-ca-4pq5w","openshift-multus/multus-w8279","kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-g59hg","openshift-dns/node-resolver-gxfnz"]
Oct 30 19:24:06.804767 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.804751    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:06.804843 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.804820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:06.806103 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.806088    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.807410 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.807389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:06.807520 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.807468    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:06.808902 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.808881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.810687 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.810670    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.810769 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.810720    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:06.812137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.812120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.813376 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.813360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.814719 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.814702    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.816200 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.816181    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.817665 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.817646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.819398 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.819382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.820924 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.820905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.860115 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.860090    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.860233 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.860179    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.860233 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.860215    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:24:06.860432 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.860419    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:24:06.860487 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.860460    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lzgjh\""
Oct 30 19:24:06.862245 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.862221    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:24:06.862359 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.862281    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.862878 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.862417    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:24:06.862878 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.862486    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6nsf6\""
Oct 30 19:24:06.863165 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.863147    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:24:06.863268 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.863183    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.863480 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.863462    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:24:06.865627 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.865609    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.865627 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.865618    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6prqp\""
Oct 30 19:24:06.865778 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.865686    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:24:06.865778 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.865718    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.866756 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866739    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:24:06.866831 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.866831 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.866907 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgm7\" (UniqueName: \"kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.866945 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.866945 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866937    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:24:06.867019 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867019 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866968    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:24:06.867019 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.866974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867019 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867066    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:06.867154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.867154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxzz\" (UniqueName: \"kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.867317 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867394    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.867512 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867570    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867587    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867622    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.867722 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rglf\" (UniqueName: \"kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.867983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.868026 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22mrm\" (UniqueName: \"kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw2sm\" (UniqueName: \"kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868397 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5kc\" (UniqueName: \"kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.868956 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.868902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmls\" (UniqueName: \"kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869017    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869033    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2m9\" (UniqueName: \"kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml52g\" (UniqueName: \"kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.869461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.869200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.909286 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.909267    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.909364 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.909322    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.910383 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910367    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.910463 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910425    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jxzqp\""
Oct 30 19:24:06.910549 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910525    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:24:06.910549 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910540    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:24:06.910707 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910530    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.910707 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910576    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:24:06.910707 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910546    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.910815 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910755    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.910815 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910769    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hthrr\""
Oct 30 19:24:06.910815 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910785    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:06.910914 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.910896    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:24:06.913180 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.913158    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:24:06.914033 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914017    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ftg5\""
Oct 30 19:24:06.914239 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914224    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2d46\""
Oct 30 19:24:06.914239 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914233    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-s4fj9\""
Oct 30 19:24:06.914322 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914245    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ndzf5\""
Oct 30 19:24:06.914464 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914451    2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:24:06.914516 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.914502    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tl5zf"
Oct 30 19:24:06.922728 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.922711    2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:06.969373 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969347    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969485 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.969485 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.969485 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.969656 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.969656 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969656 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969594    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.969656 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969637    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969700    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969704    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969740    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969746    2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.969875 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970045    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.969990    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970122    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970169    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970243 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970227    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970268    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970354    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970385    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970392    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970442    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970421    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970449    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970475    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970517    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970532    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5kc\" (UniqueName: \"kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.970693 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970641    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmls\" (UniqueName: \"kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970666    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970729    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970735    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970762    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2m9\" (UniqueName: \"kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970790    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970820    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970886    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970925    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970955    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.971495 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970956    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml52g\" (UniqueName: \"kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.970990    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971018    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971186    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971245    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971289    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgm7\" (UniqueName: \"kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971446    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971444    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971499    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.972344 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971658    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971708    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971832    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxzz\" (UniqueName: \"kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971891    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971899    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971949    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971980    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.971988    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972003    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972009    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.972053    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:06.973222 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.972113    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:07.472094257 +0000 UTC m=+2.344977055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972118    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972121    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972173    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972293    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972322    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972421    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972435    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972464    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972468    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972513    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.974106 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972525    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972698    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972759    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rglf\" (UniqueName: \"kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972842    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972874    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.972959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973015    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.974576 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.973164    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973200    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973282    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973303    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:06.973328    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:07.473294015 +0000 UTC m=+2.346176803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973363    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973404    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973409    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mrm\" (UniqueName: \"kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973480    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2sm\" (UniqueName: \"kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973739    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973787    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975323 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973867    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.973954    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.974335    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.974733    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.974815    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.974844    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.975028    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:06.975807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:06.975075    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:07.033692 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.033618    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:07.033692 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.033649    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:07.033692 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.033668    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:07.033944 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.033744    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:07.533724935 +0000 UTC m=+2.406607711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:07.036534 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036507    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5kc\" (UniqueName: \"kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:07.036686 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036555    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmls\" (UniqueName: \"kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:07.036686 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036579    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml52g\" (UniqueName: \"kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:07.036686 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036615    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgm7\" (UniqueName: \"kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:07.036686 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036615    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2m9\" (UniqueName: \"kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:07.036686 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.036667    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxzz\" (UniqueName: \"kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:07.038964 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.038938    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rglf\" (UniqueName: \"kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:07.039053 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.038948    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mrm\" (UniqueName: \"kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:07.039114 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.039093    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2sm\" (UniqueName: \"kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:07.076643 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.076620    2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:07.157084 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.157050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:07.176542 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.176519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:07.186736 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.186708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:07.191755 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.191728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:07.207028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.207007    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:07.222985 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.222953    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:07.238929 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.238906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w8279"
Oct 30 19:24:07.255808 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.255780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:07.260763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.260732    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:07.297166 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:07.297134    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983dc1c4_4111_46f2_a526_f712eb5e56bf.slice/crio-4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27 WatchSource:0}: Error finding container 4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27: Status 404 returned error can't find the container with id 4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27
Oct 30 19:24:07.297689 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:07.297660    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a2322c0b6f8492dadef7e7f75813b1.slice/crio-f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed WatchSource:0}: Error finding container f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed: Status 404 returned error can't find the container with id f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed
Oct 30 19:24:07.302628 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.302600    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:24:07.318132 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:07.318104    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16650c5b012d27eb875de166ae5ed482.slice/crio-01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168 WatchSource:0}: Error finding container 01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168: Status 404 returned error can't find the container with id 01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168
Oct 30 19:24:07.476901 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.476868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:07.477072 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.476926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:07.477072 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.477008    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:07.477072 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.477034    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:07.477072 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.477065    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:08.477049934 +0000 UTC m=+3.349932733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:07.477218 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.477080    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:08.477073287 +0000 UTC m=+3.349956059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:07.577927 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.577835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:07.578528 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.578011    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:07.578528 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.578036    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:07.578528 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.578049    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:07.578528 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:07.578110    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:08.578092292 +0000 UTC m=+3.450975065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:07.634698 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.634647    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerStarted","Data":"01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168"}
Oct 30 19:24:07.635756 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.635731    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"01a2322c0b6f8492dadef7e7f75813b1","Type":"ContainerStarted","Data":"f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed"}
Oct 30 19:24:07.636864 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.636843    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxfnz" event={"ID":"983dc1c4-4111-46f2-a526-f712eb5e56bf","Type":"ContainerStarted","Data":"4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27"}
Oct 30 19:24:07.917891 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.917776    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:07 +0000 UTC" deadline="2028-05-16 10:30:48.655085381 +0000 UTC"
Oct 30 19:24:07.917891 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:07.917836    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13527h6m40.737252252s"
Oct 30 19:24:08.070335 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.070306    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83467d2_3710_458a_8ec7_a8cc4d2c4b3f.slice/crio-5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047 WatchSource:0}: Error finding container 5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047: Status 404 returned error can't find the container with id 5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047
Oct 30 19:24:08.087508 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.087475    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322552eb_09e0_405f_b174_a7e3317605d3.slice/crio-90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290 WatchSource:0}: Error finding container 90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290: Status 404 returned error can't find the container with id 90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290
Oct 30 19:24:08.090755 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.090730    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd690fb63_2b99_4a20_8a76_4650ce687a0f.slice/crio-6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c WatchSource:0}: Error finding container 6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c: Status 404 returned error can't find the container with id 6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c
Oct 30 19:24:08.108246 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.108211    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b1c6c7_c184_4847_a6b2_9dc5371171b5.slice/crio-c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748 WatchSource:0}: Error finding container c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748: Status 404 returned error can't find the container with id c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748
Oct 30 19:24:08.117384 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.117362    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ed874a_270d_4a83_b7d3_15d3b10454f2.slice/crio-f9a5ced2d53ebea2f21c0f02c8b0bbc58056c3ebdf4804908415c9e6e3dd2e63 WatchSource:0}: Error finding container f9a5ced2d53ebea2f21c0f02c8b0bbc58056c3ebdf4804908415c9e6e3dd2e63: Status 404 returned error can't find the container with id f9a5ced2d53ebea2f21c0f02c8b0bbc58056c3ebdf4804908415c9e6e3dd2e63
Oct 30 19:24:08.125043 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.125018    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc0e467_8e63_4365_8ef7_d6cc511ca027.slice/crio-2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c WatchSource:0}: Error finding container 2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c: Status 404 returned error can't find the container with id 2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c
Oct 30 19:24:08.298535 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.298506    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95cc74a0_83aa_4d4b_9fa9_b604093c4e97.slice/crio-da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5 WatchSource:0}: Error finding container da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5: Status 404 returned error can't find the container with id da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5
Oct 30 19:24:08.357359 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:08.357329    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode407fc34_c94f_4bfe_adf1_5ec24e48de7e.slice/crio-520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116 WatchSource:0}: Error finding container 520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116: Status 404 returned error can't find the container with id 520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116
Oct 30 19:24:08.486192 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.486150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:08.486384 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.486240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:08.486462 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.486409    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:08.486462 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.486456    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:10.486443256 +0000 UTC m=+5.359326029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:08.486588 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.486467    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:08.486588 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.486545    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:10.486525241 +0000 UTC m=+5.359408020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:08.586748 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.586657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:08.587352 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.586871    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:08.587352 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.586896    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:08.587352 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.586909    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:08.587352 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.586974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:10.586953953 +0000 UTC m=+5.459836741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:08.630994 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.630954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:08.631173 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.631092    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:08.631252 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.631182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:08.631306 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.631262    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:08.631366 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.631331    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:08.631437 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:08.631415    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:08.640945 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.640913    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8279" event={"ID":"dcc0e467-8e63-4365-8ef7-d6cc511ca027","Type":"ContainerStarted","Data":"2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c"}
Oct 30 19:24:08.642979 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.642954    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xs627" event={"ID":"322552eb-09e0-405f-b174-a7e3317605d3","Type":"ContainerStarted","Data":"90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290"}
Oct 30 19:24:08.647755 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.647730    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116"}
Oct 30 19:24:08.651618 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.651595    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4pq5w" event={"ID":"95cc74a0-83aa-4d4b-9fa9-b604093c4e97","Type":"ContainerStarted","Data":"da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5"}
Oct 30 19:24:08.653184 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.653121    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g59hg" event={"ID":"02ed874a-270d-4a83-b7d3-15d3b10454f2","Type":"ContainerStarted","Data":"f9a5ced2d53ebea2f21c0f02c8b0bbc58056c3ebdf4804908415c9e6e3dd2e63"}
Oct 30 19:24:08.655641 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.655612    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerStarted","Data":"c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748"}
Oct 30 19:24:08.657103 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.657080    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhxdt" event={"ID":"d690fb63-2b99-4a20-8a76-4650ce687a0f","Type":"ContainerStarted","Data":"6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c"}
Oct 30 19:24:08.659737 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.659681    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7" event={"ID":"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f","Type":"ContainerStarted","Data":"5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047"}
Oct 30 19:24:08.919109 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.918993    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:07 +0000 UTC" deadline="2028-06-17 04:11:00.21952387 +0000 UTC"
Oct 30 19:24:08.919109 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:08.919058    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14288h46m51.300470794s"
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.504215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.504305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.504454    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.504519    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:14.504499157 +0000 UTC m=+9.377381936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.504584    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:10.504704 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.504636    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:14.504606722 +0000 UTC m=+9.377489501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:10.605363 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.604989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:10.605363 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.605221    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:10.605363 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.605244    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:10.605363 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.605258    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:10.605363 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.605323    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:14.605302653 +0000 UTC m=+9.478185428 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:10.630491 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.630409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:10.630675 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.630551    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:10.630675 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.630653    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:10.630810 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.630747    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:10.630810 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:10.630801    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:10.630928 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:10.630870    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:11.687221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:11.686237    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerStarted","Data":"5bb7a40732fff2a59bf9001e52177054c399099559fbe16ed44a43e98b68e2f2"}
Oct 30 19:24:11.695788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:11.695748    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"01a2322c0b6f8492dadef7e7f75813b1","Type":"ContainerStarted","Data":"4b82182c5b0b339c43d8c383cdc888d55d1358f48ed9fca59c5e6251e13d3927"}
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.630942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:12.631092    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.631188    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:12.631284    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.631355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:12.631478 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:12.631432    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:12.702129 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.701229    2580 generic.go:356] "Generic (PLEG): container finished" podID="16650c5b012d27eb875de166ae5ed482" containerID="5bb7a40732fff2a59bf9001e52177054c399099559fbe16ed44a43e98b68e2f2" exitCode=0
Oct 30 19:24:12.702129 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.701328    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerDied","Data":"5bb7a40732fff2a59bf9001e52177054c399099559fbe16ed44a43e98b68e2f2"}
Oct 30 19:24:12.717181 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.716158    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="7617149aa095dff42b45f37994798cc6df5a336943a961b6923706eb7a0935e8" exitCode=0
Oct 30 19:24:12.717181 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.717110    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"7617149aa095dff42b45f37994798cc6df5a336943a961b6923706eb7a0935e8"}
Oct 30 19:24:12.822324 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:12.822254    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal" podStartSLOduration=6.8222299280000005 podStartE2EDuration="6.822229928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:11.925244603 +0000 UTC m=+6.798127398" watchObservedRunningTime="2026-10-30 19:24:12.822229928 +0000 UTC m=+7.695112721"
Oct 30 19:24:13.720104 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:13.720065    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhxdt" event={"ID":"d690fb63-2b99-4a20-8a76-4650ce687a0f","Type":"ContainerStarted","Data":"24cfca8383c90ec2d84f1090636bc8d463409610b69892bb1e2bb0833ded4840"}
Oct 30 19:24:13.723038 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:13.723016    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerStarted","Data":"1b8a2bb26fb76213359bbf98b798db5092cbdb59772579a2e9aaacb3ca1b88ab"}
Oct 30 19:24:13.940192 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:13.940120    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhxdt" podStartSLOduration=2.999412371 podStartE2EDuration="7.940100982s" totalImagesPullingTime="4.940688611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:13.830606264 +0000 UTC m=+8.703489054" watchObservedRunningTime="2026-10-30 19:24:13.940100982 +0000 UTC m=+8.812983769"
Oct 30 19:24:13.940791 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:13.940762    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" podStartSLOduration=7.940755137 podStartE2EDuration="7.940755137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:13.939930262 +0000 UTC m=+8.812813057" watchObservedRunningTime="2026-10-30 19:24:13.940755137 +0000 UTC m=+8.813637932"
Oct 30 19:24:14.538389 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.538351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:14.538586 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.538436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:14.538684 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.538662    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:14.538761 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.538740    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.538719699 +0000 UTC m=+17.411602471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:14.538871 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.538860    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:14.538935 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.538910    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.538895835 +0000 UTC m=+17.411778606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:14.630690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.630656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:14.630869 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.630815    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:14.630938 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.630908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:14.631041 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.630983    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:14.631102 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.631041    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:14.631151 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.631111    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:14.639738 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.639390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:14.639738 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.639580    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:14.639738 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.639600    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:14.639738 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.639613    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:14.639738 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:14.639663    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:22.639646184 +0000 UTC m=+17.512528961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:14.739252 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.739221    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4pq5w" event={"ID":"95cc74a0-83aa-4d4b-9fa9-b604093c4e97","Type":"ContainerStarted","Data":"e27eb20bb27b2654dc10c7d120e15a79079c02c0ae606df1681b7c25892f1c0c"}
Oct 30 19:24:14.851360 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:14.848868    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4pq5w" podStartSLOduration=3.107665294 podStartE2EDuration="8.848849473s" totalImagesPullingTime="5.741184179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:14.848100937 +0000 UTC m=+9.720983731" watchObservedRunningTime="2026-10-30 19:24:14.848849473 +0000 UTC m=+9.721732267"
Oct 30 19:24:15.743501 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:15.743469    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7" event={"ID":"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f","Type":"ContainerStarted","Data":"e22b2662cee4238721cf8048733f4094f018973c374fd28bd1ce6178cbeb2299"}
Oct 30 19:24:16.262022 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.261987    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:16.263658 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.263637    2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:16.630500 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.630431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:16.630684 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.630540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:16.630684 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:16.630550    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:16.630684 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:16.630675    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:16.630887 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.630748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:16.630887 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:16.630821    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:16.751682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.751647    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gxfnz" event={"ID":"983dc1c4-4111-46f2-a526-f712eb5e56bf","Type":"ContainerStarted","Data":"d41a4f0536969fdc23c3ea83c96fd5eb8e3a9aab404040a93e2cc36ef06b0aac"}
Oct 30 19:24:16.922165 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:16.922135    2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:24:17.623156 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.622498    2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:24:16.922153927Z","UUID":"a5e0ddc7-7f79-4540-acde-21734316586b","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:24:17.624183 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.624158    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:24:17.624183 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.624185    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:24:17.755534 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.755500    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="fa06c3dc79ae2d3970289764046a627b329dc95800838f01863557ced13df9a9" exitCode=0
Oct 30 19:24:17.756006 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.755760    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"fa06c3dc79ae2d3970289764046a627b329dc95800838f01863557ced13df9a9"}
Oct 30 19:24:17.760735 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.760691    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7" event={"ID":"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f","Type":"ContainerStarted","Data":"9aa4e3ae146f6766fad65e9fad27d657a38842fbacb7985c805c46109f5be65b"}
Oct 30 19:24:17.764237 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.764210    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w8279" event={"ID":"dcc0e467-8e63-4365-8ef7-d6cc511ca027","Type":"ContainerStarted","Data":"3e8e1abadfe4cc896231f08f18ac37fef262cd7fc7da2b48809116dad0d83551"}
Oct 30 19:24:17.878533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.878413    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gxfnz" podStartSLOduration=3.495265612 podStartE2EDuration="11.878395722s" totalImagesPullingTime="8.38313011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:16.862245373 +0000 UTC m=+11.735128169" watchObservedRunningTime="2026-10-30 19:24:17.878395722 +0000 UTC m=+12.751278520"
Oct 30 19:24:17.997003 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:17.996948    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-w8279" podStartSLOduration=3.242502601 podStartE2EDuration="11.996932828s" totalImagesPullingTime="8.754430227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:17.996485474 +0000 UTC m=+12.869368276" watchObservedRunningTime="2026-10-30 19:24:17.996932828 +0000 UTC m=+12.869815621"
Oct 30 19:24:18.094554 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.094524    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:18.094748 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.094669    2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:24:18.095402 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.095376    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:18.630596 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.630508    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:18.630596 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.630556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:18.630827 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.630599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:18.630827 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:18.630666    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:18.630827 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:18.630812    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:18.630984 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:18.630902    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:18.768690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.768657    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xs627" event={"ID":"322552eb-09e0-405f-b174-a7e3317605d3","Type":"ContainerStarted","Data":"d322c7128787fd11fe5d65b6e0ad949d085ac9f7d92ae5aa01cb05bce2cf8db9"}
Oct 30 19:24:18.878138 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:18.878088    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xs627" podStartSLOduration=4.181749927 podStartE2EDuration="12.878071703s" totalImagesPullingTime="8.696321776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:18.877943239 +0000 UTC m=+13.750826030" watchObservedRunningTime="2026-10-30 19:24:18.878071703 +0000 UTC m=+13.750954497"
Oct 30 19:24:19.772872 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:19.772841    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g59hg" event={"ID":"02ed874a-270d-4a83-b7d3-15d3b10454f2","Type":"ContainerStarted","Data":"15986bd1d3f78db462eed6ad37cc73e8ef68ae173f6d498539785d67d74508fd"}
Oct 30 19:24:19.775162 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:19.775128    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="3b22b73fa4a4944d138a4ed2364df0beceb80107a9951996de3ee48ec8c7e117" exitCode=0
Oct 30 19:24:19.775285 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:19.775211    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"3b22b73fa4a4944d138a4ed2364df0beceb80107a9951996de3ee48ec8c7e117"}
Oct 30 19:24:19.777630 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:19.777603    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7" event={"ID":"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f","Type":"ContainerStarted","Data":"1b705598a173dd0891ec4d8f297c734231167b9e48c2736ecdd0ae966fa34cff"}
Oct 30 19:24:19.884735 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:19.884678    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g59hg" podStartSLOduration=2.681749987 podStartE2EDuration="13.884662438s" totalImagesPullingTime="11.202912451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:19.884515928 +0000 UTC m=+14.757398747" watchObservedRunningTime="2026-10-30 19:24:19.884662438 +0000 UTC m=+14.757545233"
Oct 30 19:24:20.116769 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:20.116673    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7" podStartSLOduration=3.789790038 podStartE2EDuration="14.116659641s" totalImagesPullingTime="10.326869603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:20.115890921 +0000 UTC m=+14.988773717" watchObservedRunningTime="2026-10-30 19:24:20.116659641 +0000 UTC m=+14.989542434"
Oct 30 19:24:20.630283 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:20.630249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:20.630283 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:20.630272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:20.630510 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:20.630293    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:20.630510 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:20.630377    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:20.630510 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:20.630469    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:20.630666 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:20.630534    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:21.785600 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:21.785550    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="9d5b98b42a65fa448b2f5f759e3ce422e44d00e3e29985b7bd24df6b058eee5f" exitCode=0
Oct 30 19:24:21.786266 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:21.785600    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"9d5b98b42a65fa448b2f5f759e3ce422e44d00e3e29985b7bd24df6b058eee5f"}
Oct 30 19:24:22.601503 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.601280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:22.601745 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.601443    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:22.601745 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.601577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:22.601745 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.601644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret podName:4d9e730e-03fe-43f4-983e-00a9e6975b6e nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.601620316 +0000 UTC m=+33.474503093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret") pod "global-pull-secret-syncer-vg85t" (UID: "4d9e730e-03fe-43f4-983e-00a9e6975b6e") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:24:22.601745 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.601682    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:22.601745 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.601737    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.601721971 +0000 UTC m=+33.474604765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:24:22.630682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.630656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:22.630682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.630672    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:22.630890 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.630798    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:22.630890 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.630840    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:22.630974 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.630940    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:22.631031 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.631011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:22.702757 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:22.702729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:22.702910 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.702892    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:24:22.702946 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.702916    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:24:22.702946 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.702925    2580 projected.go:196] Error preparing data for projected volume kube-api-access-f2vhh for pod openshift-network-diagnostics/network-check-target-cjw9x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:22.703020 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:22.702971    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh podName:c1f9acdb-b018-4e51-aab4-27a6362dc070 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.702959214 +0000 UTC m=+33.575841989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f2vhh" (UniqueName: "kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh") pod "network-check-target-cjw9x" (UID: "c1f9acdb-b018-4e51-aab4-27a6362dc070") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:24:23.791657 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:23.791625    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"70b683a2770a0571ec3e1d9c71946a4f995f10634b11dfe1289364d2f432d808"}
Oct 30 19:24:24.630702 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.630614    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:24.630702 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.630639    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:24.630702 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.630627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:24.630958 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:24.630715    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:24.630958 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:24.630806    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:24.630958 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:24.630869    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:24.796637 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.796604    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"5a294e9b292c49c466078fe87fd54a17dd540dcc112d41b32473a9b8173008d3"}
Oct 30 19:24:24.797028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.796695    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"c4bbff9f60c0ff33de536a10369a71c222f3e6aa0e7eef14cb8ac589950641b4"}
Oct 30 19:24:24.797028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.796747    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"50572b7b13c68140a605d197bfedc3f88a885b23889f791e47e2742530b29117"}
Oct 30 19:24:24.797028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.796793    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"cd00772cb7a2ac6cb51a8218d6abf3d2f448268d63d7805e021b13a2c1268c0d"}
Oct 30 19:24:24.797028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.796832    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"ffd9f255796728a84b2abd99411c309a7bfa660c254c8ee3f704d22938d432bc"}
Oct 30 19:24:24.799149 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.799123    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="a699990ddce3c38a357e26141ef7959b1cfddb11eebf01a0fe92deaef1365e41" exitCode=0
Oct 30 19:24:24.799241 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:24.799170    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"a699990ddce3c38a357e26141ef7959b1cfddb11eebf01a0fe92deaef1365e41"}
Oct 30 19:24:25.803700 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:25.803668    2580 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="88b980d48f605e65446da3bca85f8329592d59f328406f2cc09e05d12f14b90d" exitCode=0
Oct 30 19:24:25.804155 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:25.803724    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerDied","Data":"88b980d48f605e65446da3bca85f8329592d59f328406f2cc09e05d12f14b90d"}
Oct 30 19:24:26.630927 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.630890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:26.631069 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:26.631001    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:26.631069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.630890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:26.631158 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:26.631083    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:26.631158 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.630899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:26.631224 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:26.631193    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:26.808519 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.808485    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" event={"ID":"d0b1c6c7-c184-4847-a6b2-9dc5371171b5","Type":"ContainerStarted","Data":"35d36154481930bc1b3ed2cc50f4a6cfc3d7311015b9f1a9d9e6f844be4017fa"}
Oct 30 19:24:26.811641 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.811617    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"11ff403b0a2bb2674b3aaf64b87c0d8a69d0431a1dd086e4f5fd43ebef0ba2bb"}
Oct 30 19:24:26.924356 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:26.924314    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q6ssb" podStartSLOduration=7.407863142 podStartE2EDuration="20.924300042s" totalImagesPullingTime="13.5164369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:26.924267318 +0000 UTC m=+21.797150111" watchObservedRunningTime="2026-10-30 19:24:26.924300042 +0000 UTC m=+21.797182873"
Oct 30 19:24:28.631132 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.630911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:28.631678 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.630911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:28.631678 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:28.631156    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:28.631678 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.630911    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:28.631678 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:28.631233    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:28.631678 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:28.631298    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:28.818122 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.818091    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" event={"ID":"e407fc34-c94f-4bfe-adf1-5ec24e48de7e","Type":"ContainerStarted","Data":"baa2200edb85faa9639cfde71255e69d29306e8e53bcae55560a6512f3e4f114"}
Oct 30 19:24:28.818388 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.818370    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:28.818463 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.818444    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:28.833635 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:28.833610    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:29.085074 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:29.085024    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx" podStartSLOduration=7.737102134 podStartE2EDuration="23.085012776s" totalImagesPullingTime="15.347910642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:29.084612115 +0000 UTC m=+23.957494908" watchObservedRunningTime="2026-10-30 19:24:29.085012776 +0000 UTC m=+23.957895570"
Oct 30 19:24:29.821014 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:29.820983    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:29.835336 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:29.835313    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:30.631310 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.631282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:30.631502 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.631354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:30.631502 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.631458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:30.631502 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.631482    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:30.631679 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.631531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:30.631679 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.631591    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:30.824847 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.824812    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vg85t"]
Oct 30 19:24:30.825295 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.824910    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:30.825295 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.825013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:30.828528 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.828501    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lf6jh"]
Oct 30 19:24:30.828654 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.828604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:30.828701 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.828681    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:30.831661 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.831636    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjw9x"]
Oct 30 19:24:30.831747 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:30.831724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:30.831837 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:30.831817    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:32.630315 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:32.630239    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:32.630315 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:32.630261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:32.630712 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:32.630252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:32.630712 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:32.630359    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:32.630712 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:32.630409    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:32.630712 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:32.630471    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:34.630575 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:34.630530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:34.630952 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:34.630530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:34.630952 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:34.630658    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:34.630952 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:34.630709    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:34.630952 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:34.630530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:34.630952 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:34.630763    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:36.630969 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.630894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:36.631310 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.630894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:36.631310 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:36.631004    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lf6jh" podUID="1a71c312-1f67-4e38-a2dd-d54417c26e9c"
Oct 30 19:24:36.631310 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.631012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:36.631310 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:36.631066    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cjw9x" podUID="c1f9acdb-b018-4e51-aab4-27a6362dc070"
Oct 30 19:24:36.631310 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:36.631140    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vg85t" podUID="4d9e730e-03fe-43f4-983e-00a9e6975b6e"
Oct 30 19:24:36.909385 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.909174    2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeReady"
Oct 30 19:24:36.909506 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.909434    2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:24:36.993735 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.993705    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtrx2"]
Oct 30 19:24:36.995545 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:36.995530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:37.010242 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.010219    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx99q"]
Oct 30 19:24:37.011895 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.011880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:37.018663 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.018638    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldqhw"]
Oct 30 19:24:37.020425 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.020408    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"]
Oct 30 19:24:37.020554 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.020535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:37.022235 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.022215    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sltb5"]
Oct 30 19:24:37.022343 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.022315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.024014 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.023998    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"]
Oct 30 19:24:37.024100 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.024091    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.029154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.029137    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zdmh7"]
Oct 30 19:24:37.029283 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.029267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.030832 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.030817    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"]
Oct 30 19:24:37.030942 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.030928    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.032701 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.032681    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"]
Oct 30 19:24:37.033254 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.033234    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.034624 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.034608    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:24:37.034730 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.034717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.036321 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.036301    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"]
Oct 30 19:24:37.036421 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.036407    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.038319 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.038300    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"]
Oct 30 19:24:37.038406 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.038389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:37.040324 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.040303    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:24:37.040901 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.040884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.042899 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.042880    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8598d5bb4c-4xl9z"]
Oct 30 19:24:37.042972 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.042961    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.045363 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.045346    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"]
Oct 30 19:24:37.045464 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.045451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.047223 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.047208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:37.057931 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.057916    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-54xrz"]
Oct 30 19:24:37.059816 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.059796    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sh5vv"]
Oct 30 19:24:37.059900 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.059863    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.061574 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.061545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.064691 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.064674    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.064691 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.064682    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.065810 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.065791    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtrx2"]
Oct 30 19:24:37.066492 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.066475    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx99q"]
Oct 30 19:24:37.076209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.075163    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:24:37.076209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.075242    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4qk4l\""
Oct 30 19:24:37.076209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.076096    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldqhw"]
Oct 30 19:24:37.079333 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.079314    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"]
Oct 30 19:24:37.092115 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.092094    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"]
Oct 30 19:24:37.099324 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.099300    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sltb5"]
Oct 30 19:24:37.099460 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.099447    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zdmh7"]
Oct 30 19:24:37.103591 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.103351    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"]
Oct 30 19:24:37.104248 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.104229    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"]
Oct 30 19:24:37.106287 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.106263    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"]
Oct 30 19:24:37.107895 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.107873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.108036 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.107921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn899\" (UniqueName: \"kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.108036 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.107965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r7cc\" (UniqueName: \"kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc\") pod \"kas-connection-checker-685895c7c9-rtrx2\" (UID: \"45a28670-76ac-4e67-be4b-72e0a64e6766\") " pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:37.108036 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.107994    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.108036 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.108362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z8d\" (UniqueName: \"kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.108362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108119    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.108362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtdrq\" (UniqueName: \"kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq\") pod \"kas-connection-checker-685895c7c9-xx99q\" (UID: \"11477705-d48e-4ba3-a15a-8aba013e83cc\") " pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:37.108362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108200    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.108362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108217    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54xrz"]
Oct 30 19:24:37.108771 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108695    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6gk\" (UniqueName: \"kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.108771 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.108879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108786    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.108879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.108879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjdt\" (UniqueName: \"kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.109024 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmfv\" (UniqueName: \"kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.109024 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108938    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.109024 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.108971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.109024 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109170    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdx5\" (UniqueName: \"kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5\") pod \"volume-data-source-validator-58c9f5f46d-fc8s9\" (UID: \"fa0d5c4b-3127-404e-9d75-92b008cd639c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:37.109209 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109238    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmv97\" (UniqueName: \"kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109775    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9tgj\" (UniqueName: \"kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.109965 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.109920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcmd\" (UniqueName: \"kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd\") pod \"kas-connection-checker-685895c7c9-ldqhw\" (UID: \"8c679173-6012-49d3-9a8b-f1b846552a35\") " pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.110367 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.110540 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.110540 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.110422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzgk\" (UniqueName: \"kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk\") pod \"network-check-source-6f78d59f99-lskdw\" (UID: \"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:37.128090 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.128061    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"]
Oct 30 19:24:37.146607 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.146583    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.146837 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.146721    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.146837 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.146756    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.146959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.146842    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.146959 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.146870    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147079 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147063    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147072    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147082    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147071    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147083    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147105    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147114    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147137 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147139    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147297    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147337    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147337    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147379    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147383    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147393 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147378    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147399    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147539    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147553    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147542    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147613    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147626    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147639    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147648    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.147682 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.147666    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:24:37.194679 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.194647    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:24:37.204350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.204324    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:24:37.204462 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.204347    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:24:37.204860 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.204845    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:24:37.205616 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.205602    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"]
Oct 30 19:24:37.206252 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.206236    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sh5vv"]
Oct 30 19:24:37.210955 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.210939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbhf\" (UniqueName: \"kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.211001 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.210970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.211001 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.210988    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm54z\" (UniqueName: \"kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.211085 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.211182 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.211258 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.211296 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211278    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmv97\" (UniqueName: \"kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.211362 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.211419 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.211471 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211411    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tgj\" (UniqueName: \"kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.211471 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211447    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.211580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.211580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.211580 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcmd\" (UniqueName: \"kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd\") pod \"kas-connection-checker-685895c7c9-ldqhw\" (UID: \"8c679173-6012-49d3-9a8b-f1b846552a35\") " pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:37.211725 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.211725 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.211725 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211789    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211813    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzgk\" (UniqueName: \"kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk\") pod \"network-check-source-6f78d59f99-lskdw\" (UID: \"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:37.211870 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211846    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211876    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn899\" (UniqueName: \"kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211902    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7cc\" (UniqueName: \"kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc\") pod \"kas-connection-checker-685895c7c9-rtrx2\" (UID: \"45a28670-76ac-4e67-be4b-72e0a64e6766\") " pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.211972    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z8d\" (UniqueName: \"kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdrq\" (UniqueName: \"kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq\") pod \"kas-connection-checker-685895c7c9-xx99q\" (UID: \"11477705-d48e-4ba3-a15a-8aba013e83cc\") " pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212101    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212111    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.212139 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6gk\" (UniqueName: \"kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212116    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212220    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjdt\" (UniqueName: \"kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmfv\" (UniqueName: \"kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.212533 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212532    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212619    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212670    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212694    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212722    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdx5\" (UniqueName: \"kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5\") pod \"volume-data-source-validator-58c9f5f46d-fc8s9\" (UID: \"fa0d5c4b-3127-404e-9d75-92b008cd639c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:37.212843 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.212868    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.712854659 +0000 UTC m=+32.585737444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.212994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213025    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.213128 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.213350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.213350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.213350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.213350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.213327    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:37.213478 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.213372    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.713360949 +0000 UTC m=+32.586243721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213655    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.213977    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214155    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214261    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214298    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.214326 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214279    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214350    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214476    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214549    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214602    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.214690 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.214637    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.215065 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.215044    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.215167 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.215150    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.218499 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.218477    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:24:37.218725 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.218665    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zkvzq\""
Oct 30 19:24:37.218922 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.218907    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:24:37.219508 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.219496    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8598d5bb4c-4xl9z"]
Oct 30 19:24:37.220143 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.220120    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:24:37.220474 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.220442    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:24:37.220578 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.220489    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:24:37.222531 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.222515    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:24:37.222587 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.222515    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:37.222587 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.222580    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls podName:b25523fa-8b51-42e3-8e6a-79af2ce1c71c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.722555208 +0000 UTC m=+32.595438012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dkqt5" (UID: "b25523fa-8b51-42e3-8e6a-79af2ce1c71c") : secret "samples-operator-tls" not found
Oct 30 19:24:37.222662 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.222587    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:37.222662 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.222635    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.722622648 +0000 UTC m=+32.595505422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:37.222841 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.222824    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:24:37.222925 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.222841    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:24:37.222925 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.222855    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:24:37.222925 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.222847    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:24:37.222925 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.222870    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:24:37.223913 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.223900    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:37.223965 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.223914    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:37.223965 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.223948    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.723938364 +0000 UTC m=+32.596821148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:37.227415 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.227398    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.232590 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.232551    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:37.232668 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.232627    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.732616715 +0000 UTC m=+32.605499487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:37.233454 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.233433    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.234905 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.234883    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.239731 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.239708    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:24:37.240687 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240650    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:24:37.240788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240702    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-llrr6\""
Oct 30 19:24:37.240788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240743    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:24:37.240788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240756    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zgmvh\""
Oct 30 19:24:37.240788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240766    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:24:37.240788 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240780    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240797    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7vvd5\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240804    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wzkg2\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240893    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5wmd9\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240917    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-46rfm\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240931    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-58c9c\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240935    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w6qzk\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240937    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qhxtv\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240956    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9th84\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.240943    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:24:37.241041 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.241009    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f29wx\""
Oct 30 19:24:37.241925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.241911    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:37.241972 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.241956    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.741945702 +0000 UTC m=+32.614828476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:37.244143 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.244093    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.244294 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.244233    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.244364 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.244349    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.244518 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.244498    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.244861 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.244836    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.245593 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.245552    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.245775 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.245760    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.269999 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.269969    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.272239 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.272215    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjdt\" (UniqueName: \"kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.272667 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.272648    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdrq\" (UniqueName: \"kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq\") pod \"kas-connection-checker-685895c7c9-xx99q\" (UID: \"11477705-d48e-4ba3-a15a-8aba013e83cc\") " pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:37.274084 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.273998    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcmd\" (UniqueName: \"kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd\") pod \"kas-connection-checker-685895c7c9-ldqhw\" (UID: \"8c679173-6012-49d3-9a8b-f1b846552a35\") " pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:37.275069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.274577    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmv97\" (UniqueName: \"kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.275069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.274908    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tgj\" (UniqueName: \"kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.275069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.274932    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.275069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.274987    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmfv\" (UniqueName: \"kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.275069 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.275056    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7cc\" (UniqueName: \"kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc\") pod \"kas-connection-checker-685895c7c9-rtrx2\" (UID: \"45a28670-76ac-4e67-be4b-72e0a64e6766\") " pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:37.275714 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.275696    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzgk\" (UniqueName: \"kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk\") pod \"network-check-source-6f78d59f99-lskdw\" (UID: \"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:37.275944 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.275901    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6gk\" (UniqueName: \"kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.277793 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.277767    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z8d\" (UniqueName: \"kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.278312 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.278292    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.278447 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.278429    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.278536 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.278516    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn899\" (UniqueName: \"kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.278808 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.278784    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdx5\" (UniqueName: \"kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5\") pod \"volume-data-source-validator-58c9f5f46d-fc8s9\" (UID: \"fa0d5c4b-3127-404e-9d75-92b008cd639c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:37.306403 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.306382    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:37.314618 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.314706 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.314706 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.314844 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.314844 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.314782    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:24:37.314844 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbhf\" (UniqueName: \"kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.314844 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.314829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm54z\" (UniqueName: \"kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.315284 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.314851    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls podName:81a6af15-f8e9-4b72-b798-ebf0438e83f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:37.814833058 +0000 UTC m=+32.687715830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls") pod "dns-default-54xrz" (UID: "81a6af15-f8e9-4b72-b798-ebf0438e83f4") : secret "dns-default-metrics-tls" not found
Oct 30 19:24:37.315284 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.315085    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.315798 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.315768    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.317234 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.317213    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.321060 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.321042    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:37.333642 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.333620    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:37.354514 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.354481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:37.363361 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.361225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:37.367894 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.367708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:37.388157 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.387935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:37.395521 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.395350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:37.396107 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.396079    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm54z\" (UniqueName: \"kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.396538 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.396515    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbhf\" (UniqueName: \"kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.437657 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.437309    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:37.445551 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.445013    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:37.692307 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.692277    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xx99q"]
Oct 30 19:24:37.694402 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.694378    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtrx2"]
Oct 30 19:24:37.695209 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.695186    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11477705_d48e_4ba3_a15a_8aba013e83cc.slice/crio-fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1 WatchSource:0}: Error finding container fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1: Status 404 returned error can't find the container with id fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1
Oct 30 19:24:37.698636 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.698609    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a28670_76ac_4e67_be4b_72e0a64e6766.slice/crio-e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb WatchSource:0}: Error finding container e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb: Status 404 returned error can't find the container with id e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb
Oct 30 19:24:37.714285 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.714191    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-ldqhw"]
Oct 30 19:24:37.717122 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.717096    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c679173_6012_49d3_9a8b_f1b846552a35.slice/crio-3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e WatchSource:0}: Error finding container 3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e: Status 404 returned error can't find the container with id 3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e
Oct 30 19:24:37.728642 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.728612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.728735 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.728665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:37.728735 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.728697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:37.728847 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.728784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:37.728847 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.728827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:37.728993 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.728974    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:24:37.729056 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.729041    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls podName:b25523fa-8b51-42e3-8e6a-79af2ce1c71c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.72902206 +0000 UTC m=+33.601904851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dkqt5" (UID: "b25523fa-8b51-42e3-8e6a-79af2ce1c71c") : secret "samples-operator-tls" not found
Oct 30 19:24:37.729993 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.729973    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:37.730100 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.729994    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:37.730100 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.730060    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:37.730100 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.730061    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:37.730100 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.730083    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:37.730330 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.730133    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.730116447 +0000 UTC m=+33.602999236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:37.731164 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.731135    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.731115981 +0000 UTC m=+33.603998773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:37.731266 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.731208    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.731193625 +0000 UTC m=+33.604076411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:37.731712 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.731683    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.731668235 +0000 UTC m=+33.604551019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:37.745038 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.745011    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"]
Oct 30 19:24:37.749658 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.749631    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfafb0f_0c2e_4703_be40_c9c2a080fd48.slice/crio-f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b WatchSource:0}: Error finding container f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b: Status 404 returned error can't find the container with id f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b
Oct 30 19:24:37.790426 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.790394    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"]
Oct 30 19:24:37.795187 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.794109    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zdmh7"]
Oct 30 19:24:37.795187 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.794162    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"]
Oct 30 19:24:37.806586 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.806208    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f913148_3853_4762_a98a_148e2fdb44b8.slice/crio-2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418 WatchSource:0}: Error finding container 2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418: Status 404 returned error can't find the container with id 2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418
Oct 30 19:24:37.829505 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.829479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:37.829654 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.829515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:37.829712 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829658    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:24:37.829712 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.829685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:37.829807 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829716    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls podName:81a6af15-f8e9-4b72-b798-ebf0438e83f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.829702133 +0000 UTC m=+33.702584907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls") pod "dns-default-54xrz" (UID: "81a6af15-f8e9-4b72-b798-ebf0438e83f4") : secret "dns-default-metrics-tls" not found
Oct 30 19:24:37.829807 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829658    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:37.829807 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829783    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.829770745 +0000 UTC m=+33.702653530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:37.829807 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829784    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:37.829942 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:37.829835    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:38.829819939 +0000 UTC m=+33.702702724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:37.840348 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.840310    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerStarted","Data":"2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418"}
Oct 30 19:24:37.841825 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.841794    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj" event={"ID":"ecfafb0f-0c2e-4703-be40-c9c2a080fd48","Type":"ContainerStarted","Data":"f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b"}
Oct 30 19:24:37.843461 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.843433    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8" event={"ID":"68246332-14ef-4b27-b072-c6141c820f53","Type":"ContainerStarted","Data":"189d14bfd06f3c15affe7b9c62bad9b32552251c71f82a07791ad79c32f1b508"}
Oct 30 19:24:37.845165 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.845136    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rtrx2" event={"ID":"45a28670-76ac-4e67-be4b-72e0a64e6766","Type":"ContainerStarted","Data":"e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb"}
Oct 30 19:24:37.846751 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.846725    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9" event={"ID":"fa0d5c4b-3127-404e-9d75-92b008cd639c","Type":"ContainerStarted","Data":"8cd508dfd709efd272e8f94dc2bb664ad092f803c6023c0a29a6f69b84b66608"}
Oct 30 19:24:37.847860 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.847839    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldqhw" event={"ID":"8c679173-6012-49d3-9a8b-f1b846552a35","Type":"ContainerStarted","Data":"3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e"}
Oct 30 19:24:37.848957 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.848938    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xx99q" event={"ID":"11477705-d48e-4ba3-a15a-8aba013e83cc","Type":"ContainerStarted","Data":"fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1"}
Oct 30 19:24:37.854263 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.854237    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"]
Oct 30 19:24:37.858463 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.858438    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0daabe8_7547_4768_9157_55e9a70f1fc1.slice/crio-cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6 WatchSource:0}: Error finding container cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6: Status 404 returned error can't find the container with id cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6
Oct 30 19:24:37.868005 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.867984    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sh5vv"]
Oct 30 19:24:37.871257 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.871234    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8944ed8b_8a0f_485b_88e4_177f50a9f437.slice/crio-4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae WatchSource:0}: Error finding container 4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae: Status 404 returned error can't find the container with id 4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae
Oct 30 19:24:37.983855 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:37.983774    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"]
Oct 30 19:24:37.987070 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:37.987047    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf259bbb5_00a3_4a12_92e5_7ea6e2801ec5.slice/crio-bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c WatchSource:0}: Error finding container bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c: Status 404 returned error can't find the container with id bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c
Oct 30 19:24:38.631161 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.631128    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:38.631394 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.631161    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:38.631539 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.631182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:38.637841 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.637820    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:38.637957 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.637936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:38.681513 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.681487    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:24:38.681513 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.681499    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9jbqb\""
Oct 30 19:24:38.681796 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.681770    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vwkbz\""
Oct 30 19:24:38.681839 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.681806    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:24:38.681967 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.681952    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4g88n\""
Oct 30 19:24:38.688020 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.688002    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:24:38.688070 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.688063    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:25:10.688046998 +0000 UTC m=+65.560929768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : secret "metrics-daemon-secret" not found
Oct 30 19:24:38.690254 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.690230    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:38.738350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738374    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738492    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.738475103 +0000 UTC m=+35.611357876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738669    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738721    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.738707398 +0000 UTC m=+35.611590173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738674    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738754    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:38.738763 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.738757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738774    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738786    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738807    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.738784913 +0000 UTC m=+35.611667700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738832    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738842    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.738826042 +0000 UTC m=+35.611708828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:38.739193 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.738860    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls podName:b25523fa-8b51-42e3-8e6a-79af2ce1c71c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.738852392 +0000 UTC m=+35.611735163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dkqt5" (UID: "b25523fa-8b51-42e3-8e6a-79af2ce1c71c") : secret "samples-operator-tls" not found
Oct 30 19:24:38.740705 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.740679    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.840132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.840173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.840299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840418    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840464    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.840452243 +0000 UTC m=+35.713335014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840507    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840539    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.840529639 +0000 UTC m=+35.713412424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840600    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:24:38.840639 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:38.840623    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls podName:81a6af15-f8e9-4b72-b798-ebf0438e83f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:40.840616929 +0000 UTC m=+35.713499699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls") pod "dns-default-54xrz" (UID: "81a6af15-f8e9-4b72-b798-ebf0438e83f4") : secret "dns-default-metrics-tls" not found
Oct 30 19:24:38.854765 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.854729    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-ldqhw" event={"ID":"8c679173-6012-49d3-9a8b-f1b846552a35","Type":"ContainerStarted","Data":"0d056c2f17fcbfb3fbdcb5c2020476b43f5252dfdd80b525fe75f181bcd8e7c7"}
Oct 30 19:24:38.858484 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.858451    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xx99q" event={"ID":"11477705-d48e-4ba3-a15a-8aba013e83cc","Type":"ContainerStarted","Data":"0ad719fa7312c958c98293a9d078316cd62f4e7d050293b2c6fc66ab32ddc004"}
Oct 30 19:24:38.860272 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.860223    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sh5vv" event={"ID":"8944ed8b-8a0f-485b-88e4-177f50a9f437","Type":"ContainerStarted","Data":"4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae"}
Oct 30 19:24:38.862146 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.862098    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw" event={"ID":"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5","Type":"ContainerStarted","Data":"bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c"}
Oct 30 19:24:38.864345 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.864317    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" event={"ID":"b0daabe8-7547-4768-9157-55e9a70f1fc1","Type":"ContainerStarted","Data":"cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6"}
Oct 30 19:24:38.867208 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.866651    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rtrx2" event={"ID":"45a28670-76ac-4e67-be4b-72e0a64e6766","Type":"ContainerStarted","Data":"955b6f2618a91254970f9c79b1649b2194a64e430c2bd9bd74ee318710eb7539"}
Oct 30 19:24:38.945089 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.945058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:38.950266 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:38.950243    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:39.074068 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.072673    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rtrx2" podStartSLOduration=415.072655404 podStartE2EDuration="6m55.072655404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:39.071959067 +0000 UTC m=+33.944841950" watchObservedRunningTime="2026-10-30 19:24:39.072655404 +0000 UTC m=+33.945538199"
Oct 30 19:24:39.185883 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.185829    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xx99q" podStartSLOduration=415.185815561 podStartE2EDuration="6m55.185815561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:39.184702442 +0000 UTC m=+34.057585234" watchObservedRunningTime="2026-10-30 19:24:39.185815561 +0000 UTC m=+34.058698353"
Oct 30 19:24:39.296879 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.296781    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-ldqhw" podStartSLOduration=415.296762196 podStartE2EDuration="6m55.296762196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:39.296720759 +0000 UTC m=+34.169603551" watchObservedRunningTime="2026-10-30 19:24:39.296762196 +0000 UTC m=+34.169644991"
Oct 30 19:24:39.349154 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.349020    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vg85t"]
Oct 30 19:24:39.360673 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:39.360640    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9e730e_03fe_43f4_983e_00a9e6975b6e.slice/crio-e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3 WatchSource:0}: Error finding container e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3: Status 404 returned error can't find the container with id e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3
Oct 30 19:24:39.377976 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.377950    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cjw9x"]
Oct 30 19:24:39.381083 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:39.381053    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f9acdb_b018_4e51_aab4_27a6362dc070.slice/crio-925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca WatchSource:0}: Error finding container 925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca: Status 404 returned error can't find the container with id 925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca
Oct 30 19:24:39.878873 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.878755    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjw9x" event={"ID":"c1f9acdb-b018-4e51-aab4-27a6362dc070","Type":"ContainerStarted","Data":"925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca"}
Oct 30 19:24:39.885003 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:39.884975    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vg85t" event={"ID":"4d9e730e-03fe-43f4-983e-00a9e6975b6e","Type":"ContainerStarted","Data":"e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3"}
Oct 30 19:24:40.762530 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.762493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:40.762813 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.762551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:40.762813 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.762592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:40.762813 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.762661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:40.762813 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.762700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:40.763008 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.762893    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:24:40.763008 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.762956    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls podName:b25523fa-8b51-42e3-8e6a-79af2ce1c71c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.762937862 +0000 UTC m=+39.635820633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dkqt5" (UID: "b25523fa-8b51-42e3-8e6a-79af2ce1c71c") : secret "samples-operator-tls" not found
Oct 30 19:24:40.763151 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763033    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.763020749 +0000 UTC m=+39.635903537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:40.763151 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763100    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:40.763151 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763114    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:40.763151 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763146    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.763135232 +0000 UTC m=+39.636018005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:40.763410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763205    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:40.763410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763241    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.763230127 +0000 UTC m=+39.636112899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:40.763410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763300    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:40.763410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763310    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:40.763410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.763341    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.763330202 +0000 UTC m=+39.636212987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.863398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.863464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.863681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.863970    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.864082    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.864059226 +0000 UTC m=+39.736942025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.864169    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.864209    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.864198391 +0000 UTC m=+39.737081163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.864269    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:24:40.864350 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:40.864305    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls podName:81a6af15-f8e9-4b72-b798-ebf0438e83f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:44.864294042 +0000 UTC m=+39.737176816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls") pod "dns-default-54xrz" (UID: "81a6af15-f8e9-4b72-b798-ebf0438e83f4") : secret "dns-default-metrics-tls" not found
Oct 30 19:24:40.895658 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:40.894795    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9" event={"ID":"fa0d5c4b-3127-404e-9d75-92b008cd639c","Type":"ContainerStarted","Data":"e8787fe1bc4a145e99cbc366a5b0917133f043c80f5630055f07604aa1a758cf"}
Oct 30 19:24:41.006147 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:41.006061    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9" podStartSLOduration=372.178447374 podStartE2EDuration="6m15.006042617s" totalImagesPullingTime="2.827595243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:41.004992606 +0000 UTC m=+35.877875403" watchObservedRunningTime="2026-10-30 19:24:41.006042617 +0000 UTC m=+35.878925411"
Oct 30 19:24:43.909351 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.909273    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerStarted","Data":"f687ec8fb036d8858cae026d9ceb1f664321a091e1e18688de213c5243f96301"}
Oct 30 19:24:43.913028 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.913001    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj" event={"ID":"ecfafb0f-0c2e-4703-be40-c9c2a080fd48","Type":"ContainerStarted","Data":"c513e96d9132cac55b6864ba2c4a16f39cfe73b6092a3f3f3faa4ead5b656c07"}
Oct 30 19:24:43.916221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.915510    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/0.log"
Oct 30 19:24:43.916221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.915545    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0daabe8-7547-4768-9157-55e9a70f1fc1" containerID="fd266b0d319f6c34502b1f0a65a9b909f5253b3f243c581c4eaf2541463d9fbe" exitCode=255
Oct 30 19:24:43.916221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.915630    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" event={"ID":"b0daabe8-7547-4768-9157-55e9a70f1fc1","Type":"ContainerDied","Data":"fd266b0d319f6c34502b1f0a65a9b909f5253b3f243c581c4eaf2541463d9fbe"}
Oct 30 19:24:43.916221 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.915903    2580 scope.go:122] "RemoveContainer" containerID="fd266b0d319f6c34502b1f0a65a9b909f5253b3f243c581c4eaf2541463d9fbe"
Oct 30 19:24:43.918803 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:43.918765    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8" event={"ID":"68246332-14ef-4b27-b072-c6141c820f53","Type":"ContainerStarted","Data":"f76b0934ab21df34ec8959763c73335c6435d72d938c23ffbc347f1005b6eee1"}
Oct 30 19:24:44.027753 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.024431    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" podStartSLOduration=367.617277769 podStartE2EDuration="6m13.024413601s" totalImagesPullingTime="5.407135832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:44.023960658 +0000 UTC m=+38.896843451" watchObservedRunningTime="2026-10-30 19:24:44.024413601 +0000 UTC m=+38.897296396"
Oct 30 19:24:44.257695 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.257635    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj" podStartSLOduration=368.780765047 podStartE2EDuration="6m14.257617939s" totalImagesPullingTime="5.476852892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:44.256902404 +0000 UTC m=+39.129785201" watchObservedRunningTime="2026-10-30 19:24:44.257617939 +0000 UTC m=+39.130500730"
Oct 30 19:24:44.257941 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.257738    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8" podStartSLOduration=366.519854402 podStartE2EDuration="6m12.257730994s" totalImagesPullingTime="5.737876592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:44.146540925 +0000 UTC m=+39.019423719" watchObservedRunningTime="2026-10-30 19:24:44.257730994 +0000 UTC m=+39.130613792"
Oct 30 19:24:44.641631 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.641097    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qbzm8"]
Oct 30 19:24:44.670078 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.670035    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.709321 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.709289    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbzm8"]
Oct 30 19:24:44.719802 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.719778    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:24:44.753189 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.753159    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:24:44.753952 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.753931    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8x78l\""
Oct 30 19:24:44.807068 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:44.807259 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.807259 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297gw\" (UniqueName: \"kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.807259 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807187    2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 19:24:44.807259 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807211    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:44.807259 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807252    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls podName:b25523fa-8b51-42e3-8e6a-79af2ce1c71c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.807234289 +0000 UTC m=+47.680117080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dkqt5" (UID: "b25523fa-8b51-42e3-8e6a-79af2ce1c71c") : secret "samples-operator-tls" not found
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807302    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.807291542 +0000 UTC m=+47.680174318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:44.807482 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.807689 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.807499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:44.807689 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807638    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:44.807689 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807654    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:44.807781 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807703    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.80768765 +0000 UTC m=+47.680570437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:44.807781 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807742    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:44.807781 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807752    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:44.807781 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807763    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:44.807899 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807786    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.807772566 +0000 UTC m=+47.680655340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:44.807899 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.807808    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.807798003 +0000 UTC m=+47.680680792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:44.898068 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.897994    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:24:44.908092 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908245 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908116    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297gw\" (UniqueName: \"kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908245 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908245 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908419 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908265    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908419 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908312    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:44.908419 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:44.908545 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:44.908619 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908553    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:24:44.908677 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908645    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls podName:81a6af15-f8e9-4b72-b798-ebf0438e83f4 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.908626749 +0000 UTC m=+47.781509525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls") pod "dns-default-54xrz" (UID: "81a6af15-f8e9-4b72-b798-ebf0438e83f4") : secret "dns-default-metrics-tls" not found
Oct 30 19:24:44.908743 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908688    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:24:44.908743 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908719    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908748    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:45.408735849 +0000 UTC m=+40.281618636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : configmap "kube-rbac-proxy" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908762    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:45.408755359 +0000 UTC m=+40.281638131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908689    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908764    2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908790    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908815    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.908806655 +0000 UTC m=+47.781689428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:44.908925 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.908830    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.908823185 +0000 UTC m=+47.781705955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:44.909330 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.908938    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:44.916595 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.913942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:44.924657 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.924640    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:44.925145 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.925124    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/0.log"
Oct 30 19:24:44.925220 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.925161    2580 generic.go:356] "Generic (PLEG): container finished" podID="b0daabe8-7547-4768-9157-55e9a70f1fc1" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb" exitCode=255
Oct 30 19:24:44.925517 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.925500    2580 scope.go:122] "RemoveContainer" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb"
Oct 30 19:24:44.925555 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.925537    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" event={"ID":"b0daabe8-7547-4768-9157-55e9a70f1fc1","Type":"ContainerDied","Data":"1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb"}
Oct 30 19:24:44.925616 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.925604    2580 scope.go:122] "RemoveContainer" containerID="fd266b0d319f6c34502b1f0a65a9b909f5253b3f243c581c4eaf2541463d9fbe"
Oct 30 19:24:44.946817 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:44.946743    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gc2t4_openshift-console-operator(b0daabe8-7547-4768-9157-55e9a70f1fc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" podUID="b0daabe8-7547-4768-9157-55e9a70f1fc1"
Oct 30 19:24:44.960345 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.960312    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:24:44.969799 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:44.969768    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297gw\" (UniqueName: \"kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:45.009524 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.009331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.009524 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.009454    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.009817 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.009797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.009951 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.009915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.110511 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.110477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.110740 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.110661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.110828 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.110805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.112311 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.110880    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.112311 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.112054    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.112311 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.112144    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.112311 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.112164    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.172701 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.172631    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.246665 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.246643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:45.416281 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.415758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:45.416281 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.415848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:45.416281 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:45.416150    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:45.416281 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:45.416211    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:46.416192685 +0000 UTC m=+41.289075458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:45.416281 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:45.416263    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:24:45.416589 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:45.416313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:46.416301091 +0000 UTC m=+41.289183862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : configmap "kube-rbac-proxy" not found
Oct 30 19:24:45.598941 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.598914    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"]
Oct 30 19:24:45.616665 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.616635    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"]
Oct 30 19:24:45.616836 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.616721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:45.636756 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.636721    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:24:45.637100 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.636825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:45.662031 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.662007    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"]
Oct 30 19:24:45.672521 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:45.669520    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6871932b_3edd_47fd_9dcb_bc67f686b852.slice/crio-b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04 WatchSource:0}: Error finding container b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04: Status 404 returned error can't find the container with id b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04
Oct 30 19:24:45.673709 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.673684    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:24:45.673815 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.673775    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:45.673906 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.673890    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j9w96\""
Oct 30 19:24:45.719640 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.719614    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"]
Oct 30 19:24:45.720616 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.719985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwq7p\" (UniqueName: \"kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p\") pod \"migrator-66df5d496b-n78wc\" (UID: \"71223e72-c81e-4dea-ab54-53f1954babe9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:45.720616 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.720058    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bkm\" (UniqueName: \"kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm\") pod \"migrator-66df5d496b-7b6st\" (UID: \"4b032043-f55e-472f-bfae-107a66ab7aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:45.821265 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.821228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq7p\" (UniqueName: \"kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p\") pod \"migrator-66df5d496b-n78wc\" (UID: \"71223e72-c81e-4dea-ab54-53f1954babe9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:45.821446 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.821366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bkm\" (UniqueName: \"kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm\") pod \"migrator-66df5d496b-7b6st\" (UID: \"4b032043-f55e-472f-bfae-107a66ab7aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:45.879941 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.879905    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bkm\" (UniqueName: \"kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm\") pod \"migrator-66df5d496b-7b6st\" (UID: \"4b032043-f55e-472f-bfae-107a66ab7aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:45.879941 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.879928    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq7p\" (UniqueName: \"kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p\") pod \"migrator-66df5d496b-n78wc\" (UID: \"71223e72-c81e-4dea-ab54-53f1954babe9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:45.931177 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.931084    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sh5vv" event={"ID":"8944ed8b-8a0f-485b-88e4-177f50a9f437","Type":"ContainerStarted","Data":"6d4af463928876272eaea48af0daf54a3b608e07f361a79ced8976ffbfca2a13"}
Oct 30 19:24:45.932889 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.932858    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw" event={"ID":"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5","Type":"ContainerStarted","Data":"15f6887a4f3dd764e2acc71e5078a3197b4f618296dd0a52043c2ba063faafaa"}
Oct 30 19:24:45.934751 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.934732    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:45.935084 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.935069    2580 scope.go:122] "RemoveContainer" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb"
Oct 30 19:24:45.935240 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:45.935220    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gc2t4_openshift-console-operator(b0daabe8-7547-4768-9157-55e9a70f1fc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" podUID="b0daabe8-7547-4768-9157-55e9a70f1fc1"
Oct 30 19:24:45.936633 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.936606    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cjw9x" event={"ID":"c1f9acdb-b018-4e51-aab4-27a6362dc070","Type":"ContainerStarted","Data":"9602962c62c7a59d8aba297ba96cbda13584386be2925226362640e600a9d710"}
Oct 30 19:24:45.936749 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.936672    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:45.938143 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.938118    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2md8m-pt29t" event={"ID":"6871932b-3edd-47fd-9dcb-bc67f686b852","Type":"ContainerStarted","Data":"65c6d4312d3a632d1f80acaa2c3cb877db6965d15d8ea1e388859e3f28dd422c"}
Oct 30 19:24:45.938252 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.938154    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2md8m-pt29t" event={"ID":"6871932b-3edd-47fd-9dcb-bc67f686b852","Type":"ContainerStarted","Data":"b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04"}
Oct 30 19:24:45.951639 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.951617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:45.969789 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:45.969768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:46.054418 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.054368    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sh5vv" podStartSLOduration=2.7552249250000003 podStartE2EDuration="10.054355319s" totalImagesPullingTime="7.299130394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.053516339 +0000 UTC m=+40.926399125" watchObservedRunningTime="2026-10-30 19:24:46.054355319 +0000 UTC m=+40.927238109"
Oct 30 19:24:46.125587 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.125528    2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nsjps"]
Oct 30 19:24:46.162840 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.162813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.168049 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.167996    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2md8m-pt29t" podStartSLOduration=2.167977647 podStartE2EDuration="2.167977647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.16577476 +0000 UTC m=+41.038657553" watchObservedRunningTime="2026-10-30 19:24:46.167977647 +0000 UTC m=+41.040860442"
Oct 30 19:24:46.182821 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.182761    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nsjps"]
Oct 30 19:24:46.215676 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.215652    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:24:46.215807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.215730    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:24:46.215807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.215762    2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t58hx\""
Oct 30 19:24:46.215807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.215763    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:24:46.215807 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.215770    2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:24:46.327511 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.327430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd74g\" (UniqueName: \"kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.327511 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.327507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.327810 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.327712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.340138 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.340106    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"]
Oct 30 19:24:46.344024 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:46.343988    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71223e72_c81e_4dea_ab54_53f1954babe9.slice/crio-7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a WatchSource:0}: Error finding container 7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a: Status 404 returned error can't find the container with id 7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a
Oct 30 19:24:46.377171 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.377147    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"]
Oct 30 19:24:46.379655 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:46.379632    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b032043_f55e_472f_bfae_107a66ab7aa0.slice/crio-42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633 WatchSource:0}: Error finding container 42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633: Status 404 returned error can't find the container with id 42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633
Oct 30 19:24:46.395336 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.395296    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cjw9x" podStartSLOduration=34.32759093 podStartE2EDuration="40.39527925s" totalImagesPullingTime="6.06768832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.394157464 +0000 UTC m=+41.267040257" watchObservedRunningTime="2026-10-30 19:24:46.39527925 +0000 UTC m=+41.268162043"
Oct 30 19:24:46.429178 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.429148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:46.429298 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.429232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:46.429298 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:46.429244    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:24:46.429410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:46.429304    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:48.429288291 +0000 UTC m=+43.302171077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : configmap "kube-rbac-proxy" not found
Oct 30 19:24:46.429410 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.429318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.429410 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:46.429364    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:46.429410 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.429399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd74g\" (UniqueName: \"kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.429600 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:46.429431    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:48.429413343 +0000 UTC m=+43.302296136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:46.429600 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.429485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.430150 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.430127    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.432138 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.432099    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.492085 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.492021    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd74g\" (UniqueName: \"kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.510486 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.510437    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw" podStartSLOduration=367.058622715 podStartE2EDuration="6m14.510424534s" totalImagesPullingTime="7.451801819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:46.509265174 +0000 UTC m=+41.382147967" watchObservedRunningTime="2026-10-30 19:24:46.510424534 +0000 UTC m=+41.383307327"
Oct 30 19:24:46.784846 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.784757    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:46.943055 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.942961    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc" event={"ID":"71223e72-c81e-4dea-ab54-53f1954babe9","Type":"ContainerStarted","Data":"7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a"}
Oct 30 19:24:46.944525 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:46.944482    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st" event={"ID":"4b032043-f55e-472f-bfae-107a66ab7aa0","Type":"ContainerStarted","Data":"42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633"}
Oct 30 19:24:47.180064 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.179986    2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nsjps"]
Oct 30 19:24:47.182136 ip-10-0-134-200 kubenswrapper[2580]: W1030 19:24:47.182111    2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a88398_db0d_46cd_8cbb_ac92a2f2eb9f.slice/crio-3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b WatchSource:0}: Error finding container 3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b: Status 404 returned error can't find the container with id 3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b
Oct 30 19:24:47.396658 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.396623    2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:47.396825 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.396688    2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:47.397153 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.397129    2580 scope.go:122] "RemoveContainer" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb"
Oct 30 19:24:47.397387 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:47.397363    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-gc2t4_openshift-console-operator(b0daabe8-7547-4768-9157-55e9a70f1fc1)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" podUID="b0daabe8-7547-4768-9157-55e9a70f1fc1"
Oct 30 19:24:47.954855 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.954187    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps" event={"ID":"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f","Type":"ContainerStarted","Data":"6ed57cecb320090bf2e1636f457fb5c7cb49d48abbee3ea52344440417f258b7"}
Oct 30 19:24:47.954855 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:47.954243    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps" event={"ID":"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f","Type":"ContainerStarted","Data":"3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b"}
Oct 30 19:24:48.452509 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:48.452471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:48.452673 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:48.452659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:48.452740 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:48.452674    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:48.452786 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:48.452750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.452728535 +0000 UTC m=+47.325611329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:48.452786 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:48.452766    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:24:48.452879 ip-10-0-134-200 kubenswrapper[2580]: E1030 19:24:48.452810    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.452796483 +0000 UTC m=+47.325679254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : configmap "kube-rbac-proxy" not found
Oct 30 19:24:48.824164 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:48.824091    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gxfnz_983dc1c4-4111-46f2-a526-f712eb5e56bf/dns-node-resolver/0.log"
Oct 30 19:24:48.963114 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:48.963071    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc" event={"ID":"71223e72-c81e-4dea-ab54-53f1954babe9","Type":"ContainerStarted","Data":"3d420d2e03275484a48156d0cac5c7ff6abb207c71626ad25ca86ab726a09eee"}
Oct 30 19:24:48.966390 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:48.966331    2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st" event={"ID":"4b032043-f55e-472f-bfae-107a66ab7aa0","Type":"ContainerStarted","Data":"40138395f74a87161e82799413eb6dfcc499b5674d5ab77f6bf53c523915802c"}
Oct 30 19:24:49.076796 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:49.075715    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps" podStartSLOduration=3.075694499 podStartE2EDuration="3.075694499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:48.065951109 +0000 UTC m=+42.938833908" watchObservedRunningTime="2026-10-30 19:24:49.075694499 +0000 UTC m=+43.948577294"
Oct 30 19:24:49.221092 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:49.221020    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc" podStartSLOduration=2.058192905 podStartE2EDuration="4.220999925s" totalImagesPullingTime="2.16280702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:49.110170814 +0000 UTC m=+43.983053620" watchObservedRunningTime="2026-10-30 19:24:49.220999925 +0000 UTC m=+44.093882721"
Oct 30 19:24:49.221874 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:49.221840    2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st" podStartSLOduration=2.091194988 podStartE2EDuration="4.221825481s" totalImagesPullingTime="2.130630493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:49.21994972 +0000 UTC m=+44.092832512" watchObservedRunningTime="2026-10-30 19:24:49.221825481 +0000 UTC m=+44.094708275"
Oct 30 19:24:49.615501 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:49.615472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4pq5w_95cc74a0-83aa-4d4b-9fa9-b604093c4e97/node-ca/0.log"
Oct 30 19:24:50.415303 ip-10-0-134-200 kubenswrapper[2580]: I1030 19:24:50.415213    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sh5vv_8944ed8b-8a0f-485b-88e4-177f50a9f437/serve-healthcheck-canary/0.log"
Oct 30 19:24:50.424274 ip-10-0-134-200 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:24:50.442955 ip-10-0-134-200 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:24:50.443177 ip-10-0-134-200 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:24:50.443242 ip-10-0-134-200 systemd[1]: kubelet.service: Consumed 3.314s CPU time.
Oct 30 19:24:50.454280 ip-10-0-134-200 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:24:50.536610 ip-10-0-134-200 kubenswrapper[5085]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:50.536610 ip-10-0-134-200 kubenswrapper[5085]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:24:50.536610 ip-10-0-134-200 kubenswrapper[5085]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:50.536610 ip-10-0-134-200 kubenswrapper[5085]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:24:50.538687 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538600    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.538687 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538686    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.538687 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538690    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538694    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538697    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538700    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538703    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538705    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538708    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538711    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538714    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538717    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538720    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538723    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538737    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538740    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538742    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538745    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538748    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538752    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538757    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.538786 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538761    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538765    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538768    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538771    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538774    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538777    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538780    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538783    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538785    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538788    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538799    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538802    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538805    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538808    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538810    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538813    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538816    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538818    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538822    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.539260 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538825    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538827    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538830    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538833    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538835    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538838    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538840    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538849    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538852    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538855    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538858    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538861    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538864    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538867    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538869    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538876    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538879    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538882    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538885    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.539756 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538887    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538890    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538892    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538895    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538900    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538904    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538908    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538912    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538916    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538921    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538925    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538929    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538932    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538935    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538937    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538941    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538943    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538946    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538948    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538951    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.540229 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538953    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538962    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538966    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538969    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538971    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538974    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538977    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538979    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538982    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538985    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538989    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538993    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538996    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.538999    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539001    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539004    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539007    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539009    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539012    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.540734 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539014    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539017    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539019    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539022    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539024    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539027    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539029    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539032    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539890    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539901    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539906    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539909    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539912    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539915    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539917    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539923    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539926    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539929    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539932    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.541211 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539935    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539937    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539940    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539943    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539946    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539948    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539951    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539954    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539956    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539959    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539962    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539964    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539967    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539970    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539973    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539975    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539978    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539981    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539984    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539986    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.541701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539989    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539991    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539994    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539997    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.539999    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540002    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540005    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540007    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540010    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540013    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540016    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540018    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540021    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540024    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540027    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540030    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540033    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540035    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540038    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540040    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.542191 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540043    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540045    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540048    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540050    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540053    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540055    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540058    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540061    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540064    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540066    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540069    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540072    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540075    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540080    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540083    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540086    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540088    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540091    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540094    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.542694 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540096    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540099    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540102    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540105    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540107    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540110    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540112    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540116    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540119    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540121    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540124    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540126    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540129    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540132    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540135    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540137    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540140    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540143    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540146    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.543204 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540148    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540151    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540154    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540156    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540159    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540162    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540164    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540167    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540169    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540172    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540175    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540178    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540180    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540183    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540186    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540188    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540191    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540715    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540723    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.543700 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540727    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540731    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540735    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540739    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540742    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540745    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540748    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540751    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540754    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540757    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540759    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540762    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540764    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540767    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540769    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540772    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540775    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540777    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540780    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540783    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.544167 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540786    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540789    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540791    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540794    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540796    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540799    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540802    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540805    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540807    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540810    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540813    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540816    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540819    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540822    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540824    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540827    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540829    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540832    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540835    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540837    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.544706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540840    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540842    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540845    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540847    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540850    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540852    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540855    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540857    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540860    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540863    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540865    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540868    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540871    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540876    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540879    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540881    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540884    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540887    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540890    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540893    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.545297 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540895    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540898    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540901    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540904    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540906    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540909    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540912    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540914    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540917    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540920    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540923    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540925    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540928    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540930    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540933    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540936    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540938    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540941    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540943    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.546093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540946    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540948    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540951    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540954    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540956    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540959    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540964    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540967    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540969    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540972    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540974    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540978    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540982    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540986    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540989    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540992    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540995    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.540998    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541002    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.546735 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541006    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541008    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541011    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541013    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541016    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541019    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541110    5085 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541117    5085 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541124    5085 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541128    5085 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541133    5085 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541136    5085 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541140    5085 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541144    5085 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541147    5085 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541150    5085 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541154    5085 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541158    5085 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541161    5085 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541166    5085 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541169    5085 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541172    5085 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541175    5085 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:24:50.547215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541179    5085 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541183    5085 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541186    5085 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541189    5085 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541193    5085 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541196    5085 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541200    5085 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541203    5085 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541207    5085 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541210    5085 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541214    5085 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541217    5085 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541220    5085 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541223    5085 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541226    5085 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541230    5085 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541234    5085 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541237    5085 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541240    5085 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541243    5085 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541247    5085 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541251    5085 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541255    5085 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541258    5085 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:24:50.547830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541261    5085 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541264    5085 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541268    5085 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541271    5085 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541275    5085 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541278    5085 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541282    5085 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541285    5085 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541288    5085 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541291    5085 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541294    5085 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541297    5085 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541300    5085 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541305    5085 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541310    5085 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541315    5085 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541319    5085 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541325    5085 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541330    5085 flags.go:64] FLAG: --help="false"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541333    5085 flags.go:64] FLAG: --hostname-override="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541336    5085 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541339    5085 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541342    5085 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:24:50.548403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541345    5085 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541349    5085 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541352    5085 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541354    5085 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541357    5085 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541360    5085 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541363    5085 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541367    5085 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541370    5085 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541373    5085 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541376    5085 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541379    5085 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541382    5085 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541385    5085 flags.go:64] FLAG: --lock-file=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541388    5085 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541391    5085 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541394    5085 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541400    5085 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541402    5085 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541406    5085 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541409    5085 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541412    5085 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541415    5085 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541418    5085 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541421    5085 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:24:50.548971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541425    5085 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541428    5085 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541433    5085 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541436    5085 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541439    5085 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541442    5085 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541445    5085 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541448    5085 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541451    5085 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541454    5085 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541460    5085 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541465    5085 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541468    5085 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541471    5085 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541474    5085 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541477    5085 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541480    5085 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541483    5085 flags.go:64] FLAG: --port="10250"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541486    5085 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541489    5085 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e02bbff0b14edfa8"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541492    5085 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541496    5085 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541499    5085 flags.go:64] FLAG: --register-node="true"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541501    5085 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541505    5085 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541508    5085 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:24:50.549592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541511    5085 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541514    5085 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541518    5085 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541521    5085 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541524    5085 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541527    5085 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541530    5085 flags.go:64] FLAG: --runonce="false"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541533    5085 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541536    5085 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541540    5085 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541543    5085 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541546    5085 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541549    5085 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541552    5085 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541555    5085 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541570    5085 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541573    5085 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541578    5085 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541581    5085 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541584    5085 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541589    5085 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541592    5085 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541598    5085 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541601    5085 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541604    5085 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:24:50.550308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541608    5085 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541611    5085 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541614    5085 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541617    5085 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541620    5085 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541623    5085 flags.go:64] FLAG: --v="2"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541627    5085 flags.go:64] FLAG: --version="false"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541632    5085 flags.go:64] FLAG: --vmodule=""
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541636    5085 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.541639    5085 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541774    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541777    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541780    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541783    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541786    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541789    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541792    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541795    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541798    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541800    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541803    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541806    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.550950 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541808    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541811    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541815    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541818    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541821    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541824    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541827    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541829    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541832    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541835    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541838    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541840    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541843    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541846    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541848    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541851    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541855    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541859    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541862    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541864    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.551706 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541867    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541870    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541872    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541875    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541877    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541880    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541883    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541885    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541888    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541891    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541893    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541896    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541899    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541901    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541906    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541908    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541911    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541915    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541918    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541920    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.552257 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541923    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541925    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541928    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541931    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541933    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541936    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541939    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541941    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541944    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541947    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541949    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541952    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541955    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541957    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541961    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541965    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541968    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541970    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541973    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.552865 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541976    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541978    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541981    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541984    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541987    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541989    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541992    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541996    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.541999    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542001    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542005    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542008    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542011    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542013    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542016    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542018    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542021    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542024    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542026    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.553398 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542029    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542032    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542035    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542037    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542040    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542042    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542045    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542047    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542050    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542052    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542055    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542057    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542060    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542062    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542065    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.553881 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542067    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.542094    5085 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542129    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542134    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542137    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542140    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542144    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542146    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542149    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542153    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542156    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542158    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542161    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542163    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542166    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542169    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.554242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542172    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542174    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542177    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542180    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542183    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542186    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542188    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542191    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542193    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542196    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542199    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542202    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542205    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542207    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542210    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542212    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542215    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542218    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542588    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.554717 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542594    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542599    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542604    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542607    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542611    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542615    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542619    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542622    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542626    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542629    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542632    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542634    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542637    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542640    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542644    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542646    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542649    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542651    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542654    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.555331 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542656    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542659    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542662    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542664    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542667    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542670    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542672    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542675    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542678    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542680    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542683    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542685    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542688    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542691    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542694    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542698    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542701    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542704    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542706    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.555840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542710    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542713    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542716    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542718    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542721    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542723    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542726    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542728    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542731    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542733    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542736    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542739    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542741    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542744    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542747    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542749    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542752    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542755    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542757    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542760    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.556342 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542763    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542765    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542768    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542770    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542773    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542775    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542778    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542781    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542786    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542789    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542798    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542801    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542805    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542808    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.542812    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.547475    5085 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.547492    5085 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.547511    5085 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.547515    5085 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:24:50.556840 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547594    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547599    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547603    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547606    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547609    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547612    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547615    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547617    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547620    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547623    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547625    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547628    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547631    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547634    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547636    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547639    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547642    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547644    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547647    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.557289 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547650    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547653    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547656    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547659    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547661    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547664    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547667    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547670    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547673    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547676    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547678    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547681    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547684    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547686    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547689    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547691    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547694    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547697    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547700    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547702    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.557765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547705    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547707    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547710    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547712    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547715    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547718    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547722    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547727    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547730    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547733    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547735    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547738    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547740    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547744    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547746    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547749    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547752    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547754    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547758    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.558230 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547762    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547765    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547768    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547771    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547774    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547777    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547780    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547783    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547786    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547789    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547792    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547794    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547797    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547800    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547802    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547805    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547808    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547810    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547813    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.558883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547816    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547818    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547821    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547824    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547827    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547830    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547832    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547835    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547839    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547841    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547844    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547847    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547850    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547852    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547855    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547858    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547860    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547863    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547865    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.559338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547868    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547870    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547873    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547876    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547878    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547881    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547883    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547886    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547888    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.547891    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.547917    5085 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548077    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548082    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548085    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548089    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548091    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548095    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.560065 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548098    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548101    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548104    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548107    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548109    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548112    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548115    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548117    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548120    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548123    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548125    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548128    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548131    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548133    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548136    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548138    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548141    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548144    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548147    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548149    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.560697 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548152    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548154    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548157    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548159    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548162    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548164    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548167    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548170    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548172    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548175    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548177    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548179    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548183    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548185    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548188    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548191    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548193    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548196    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548198    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548201    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.561508 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548204    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548206    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548209    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548211    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548214    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548217    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548219    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548222    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548225    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548227    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548231    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548235    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548238    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548241    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548244    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548247    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548249    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548252    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548254    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.562150 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548257    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548260    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548262    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548265    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548267    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548270    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548273    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548275    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548278    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548281    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548283    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548286    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548289    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548291    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548294    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548297    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548300    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548302    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548305    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548307    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.563027 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548310    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548312    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548315    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548318    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548320    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548323    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548326    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548328    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548333    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548336    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548338    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548341    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548343    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548346    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548350    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548353    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548356    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548359    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548361    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.564008 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548365    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548369    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.548392    5085 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548422    5085 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548427    5085 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548430    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548432    5085 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548435    5085 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548438    5085 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548440    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548443    5085 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548445    5085 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548448    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548451    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548453    5085 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:24:50.564860 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548457    5085 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548461    5085 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548464    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548467    5085 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548470    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548473    5085 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548477    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548479    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548482    5085 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548485    5085 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548488    5085 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548490    5085 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548493    5085 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548495    5085 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548498    5085 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548501    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548503    5085 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548506    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548509    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:24:50.565583 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548513    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548516    5085 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548519    5085 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548522    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548525    5085 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548527    5085 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548530    5085 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548532    5085 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548535    5085 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548538    5085 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548541    5085 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548543    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548546    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548549    5085 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548551    5085 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548554    5085 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548557    5085 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548573    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548578    5085 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:24:50.566507 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548583    5085 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548586    5085 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548588    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548591    5085 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548594    5085 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548598    5085 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548602    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548606    5085 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548609    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548612    5085 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548615    5085 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548618    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548621    5085 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548625    5085 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548628    5085 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548631    5085 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548633    5085 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548636    5085 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548639    5085 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:24:50.567392 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548641    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548644    5085 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548646    5085 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548649    5085 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548652    5085 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548654    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548656    5085 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548659    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548662    5085 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548664    5085 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548667    5085 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548669    5085 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548672    5085 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548676    5085 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548679    5085 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548681    5085 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548684    5085 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548686    5085 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548689    5085 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548692    5085 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:24:50.568422 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548694    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548697    5085 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548699    5085 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548702    5085 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548705    5085 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548707    5085 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548711    5085 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548714    5085 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548716    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548719    5085 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548721    5085 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548724    5085 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548727    5085 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548730    5085 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548733    5085 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548735    5085 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:50.548738    5085 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.548794    5085 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.549735    5085 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:24:50.569267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.549786    5085 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.549961    5085 server.go:1014] "Starting client certificate rotation"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.550099    5085 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.550150    5085 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:19:06 +0000 UTC" deadline="2028-05-03 06:10:37.971904149 +0000 UTC"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.550179    5085 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13210h45m47.421726643s"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.550389    5085 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.551595    5085 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.552816    5085 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.554520    5085 log.go:25] "Validated CRI v1 image API"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.555310    5085 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:24:50.569864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.557264    5085 fs.go:135] Filesystem UUIDs: map[2ca204d7-d8fe-495c-a6fa-ac988d898c9a:/dev/nvme0n1p3 7695ae5c-5d6c-44a0-a762-5b94a71d3819:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:24:50.570171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.557284    5085 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/189d14bfd06f3c15affe7b9c62bad9b32552251c71f82a07791ad79c32f1b508/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/189d14bfd06f3c15affe7b9c62bad9b32552251c71f82a07791ad79c32f1b508/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cd508dfd709efd272e8f94dc2bb664ad092f803c6023c0a29a6f69b84b66608/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cd508dfd709efd272e8f94dc2bb664ad092f803c6023c0a29a6f69b84b66608/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~projected/kube-api-access-8mxzz:{mountpoint:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~projected/kube-api-access-8mxzz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/kube-api-access-7w26d:{mountpoint:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/kube-api-access-7w26d major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f9c4542-7226-437e-85a9-7d7639ffbab7/volumes/kubernetes.io~projected/kube-api-access-297gw:{mountpoint:/var/lib/kubelet/pods/0f9c4542-7226-437e-85a9-7d7639ffbab7/volumes/kubernetes.io~projected/kube-api-access-297gw major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11477705-d48e-4ba3-a15a-8aba013e83cc/volumes/kubernetes.io~projected/kube-api-access-rtdrq:{mountpoint:/var/lib/kubelet/pods/11477705-d48e-4ba3-a15a-8aba013e83cc/volumes/kubernetes.io~projected/kube-api-access-rtdrq major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a71c312-1f67-4e38-a2dd-d54417c26e9c/volumes/kubernetes.io~projected/kube-api-access-22mrm:{mountpoint:/var/lib/kubelet/pods/1a71c312-1f67-4e38-a2dd-d54417c26e9c/volumes/kubernetes.io~projected/kube-api-access-22mrm major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~projected/kube-api-access-x9tgj:{mountpoint:/var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~projected/kube-api-access-x9tgj major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/322552eb-09e0-405f-b174-a7e3317605d3/volumes/kubernetes.io~projected/kube-api-access-mw2sm:{mountpoint:/var/lib/kubelet/pods/322552eb-09e0-405f-b174-a7e3317605d3/volumes/kubernetes.io~projected/kube-api-access-mw2sm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45a28670-76ac-4e67-be4b-72e0a64e6766/volumes/kubernetes.io~projected/kube-api-access-2r7cc:{mountpoint:/var/lib/kubelet/pods/45a28670-76ac-4e67-be4b-72e0a64e6766/volumes/kubernetes.io~projected/kube-api-access-2r7cc major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b032043-f55e-472f-bfae-107a66ab7aa0/volumes/kubernetes.io~projected/kube-api-access-q2bkm:{mountpoint:/var/lib/kubelet/pods/4b032043-f55e-472f-bfae-107a66ab7aa0/volumes/kubernetes.io~projected/kube-api-access-q2bkm major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d9e730e-03fe-43f4-983e-00a9e6975b6e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4d9e730e-03fe-43f4-983e-00a9e6975b6e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7/volumes/kubernetes.io~projected/kube-api-access-pmv97:{mountpoint:/var/lib/kubelet/pods/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7/volumes/kubernetes.io~projected/kube-api-access-pmv97 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~projected/kube-api-access-bnjdt:{mountpoint:/var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~projected/kube-api-access-bnjdt major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6871932b-3edd-47fd-9dcb-bc67f686b852/volumes/kubernetes.io~projected/kube-api-access-jvw87:{mountpoint:/var/lib/kubelet/pods/6871932b-3edd-47fd-9dcb-bc67f686b852/volumes/kubernetes.io~projected/kube-api-access-jvw87 major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71223e72-c81e-4dea-ab54-53f1954babe9/volumes/kubernetes.io~projected/kube-api-access-bwq7p:{mountpoint:/var/lib/kubelet/pods/71223e72-c81e-4dea-ab54-53f1954babe9/volumes/kubernetes.io~projected/kube-api-access-bwq7p major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81a6af15-f8e9-4b72-b798-ebf0438e83f4/volumes/kubernetes.io~projected/kube-api-access-mm54z:{mountpoint:/var/lib/kubelet/pods/81a6af15-f8e9-4b72-b798-ebf0438e83f4/volumes/kubernetes.io~projected/kube-api-access-mm54z major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~projected/kube-api-access-rwbhf:{mountpoint:/var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~projected/kube-api-access-rwbhf major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c679173-6012-49d3-9a8b-f1b846552a35/volumes/kubernetes.io~projected/kube-api-access-njcmd:{mountpoint:/var/lib/kubelet/pods/8c679173-6012-49d3-9a8b-f1b846552a35/volumes/kubernetes.io~projected/kube-api-access-njcmd major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95cc74a0-83aa-4d4b-9fa9-b604093c4e97/volumes/kubernetes.io~projected/kube-api-access-4rglf:{mountpoint:/var/lib/kubelet/pods/95cc74a0-83aa-4d4b-9fa9-b604093c4e97/volumes/kubernetes.io~projected/kube-api-access-4rglf major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/983dc1c4-4111-46f2-a526-f712eb5e56bf/volumes/kubernetes.io~projected/kube-api-access-hfmls:{mountpoint:/var/lib/kubelet/pods/983dc1c4-4111-46f2-a526-f712eb5e56bf/volumes/kubernetes.io~projected/kube-api-access-hfmls major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~projected/kube-api-access-rn899:{mountpoint:/var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~projected/kube-api-access-rn899 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25523fa-8b51-42e3-8e6a-79af2ce1c71c/volumes/kubernetes.io~projected/kube-api-access-b9z8d:{mountpoint:/var/lib/kubelet/pods/b25523fa-8b51-42e3-8e6a-79af2ce1c71c/volumes/kubernetes.io~projected/kube-api-access-b9z8d major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1f9acdb-b018-4e51-aab4-27a6362dc070/volumes/kubernetes.io~projected/kube-api-access-f2vhh:{mountpoint:/var/lib/kubelet/pods/c1f9acdb-b018-4e51-aab4-27a6362dc070/volumes/kubernetes.io~projected/kube-api-access-f2vhh major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0b1c6c7-c184-4847-a6b2-9dc5371171b5/volumes/kubernetes.io~projected/kube-api-access-8bgm7:{mountpoint:/var/lib/kubelet/pods/d0b1c6c7-c184-4847-a6b2-9dc5371171b5/volumes/kubernetes.io~projected/kube-api-access-8bgm7 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d690fb63-2b99-4a20-8a76-4650ce687a0f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d690fb63-2b99-4a20-8a76-4650ce687a0f/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f/volumes/kubernetes.io~projected/kube-api-access-nc5kc:{mountpoint:/var/lib/kubelet/pods/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f/volumes/kubernetes.io~projected/kube-api-access-nc5kc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~projected/kube-api-access-mxmfv:{mountpoint:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~projected/kube-api-access-mxmfv major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/default-certificate major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcc0e467-8e63-4365-8ef7-d6cc511ca027/volumes/kubernetes.io~projected/kube-api-access-ml52g:{mountpoint:/var/lib/kubelet/pods/dcc0e467-8e63-4365-8ef7-d6cc511ca027/volumes/kubernetes.io~projected/kube-api-access-ml52g major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~projected/kube-api-access-gd74g:{mountpoint:/var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~projected/kube-api-access-gd74g major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~projected/kube-api-access-pn2m9:{mountpoint:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~projected/kube-api-access-pn2m9 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~projected/kube-api-access-wr6gk:{mountpoint:/var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~projected/kube-api-access-wr6gk major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5/volumes/kubernetes.io~projected/kube-api-access-5fzgk:{mountpoint:/var/lib/kubelet/pods/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5/volumes/kubernetes.io~projected/kube-api-access-5fzgk major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa0d5c4b-3127-404e-9d75-92b008cd639c/volumes/kubernetes.io~projected/kube-api-access-rvdx5:{mountpoint:/var/lib/kubelet/pods/fa0d5c4b-3127-404e-9d75-92b008cd639c/volumes/kubernetes.io~projected/kube-api-access-rvdx5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/kube-api-access-m5qr2:{mountpoint:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/kube-api-access-m5qr2 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4903bb9e01fca20eb7175bef79498a6a4e2bb6876f6137048c2904cfa5af3a82/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/24d79fa3c3ae4e7abaf37f9e0b31131d90967f19439a80e55c7b1be526aa81fd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d1cbf6b11893fc01f42e1cca3bf852b5fbada97e1e1f5f9fd9e6fd84f5414dd4/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2c410289877ca94b42e715c3062cf6e6864e9fa9875db7ddc624919327c8fc37/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a178638f827cb119bbf8b446a72a38d3a936c36b582f310be73d78159ac9d837/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a8333368aa77f1c48bc1c41d9f321e841f6d579a0be40f73e0f1b95181fa2ea0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/20f5d7999618b21fae01235f8ff4fd97ac42305d6dea21cbf766d89a6048dd13/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/49e6ca5fe89db574408d2cbf63071ef43a5cee4d20669fb9ada33f6bd12d6322/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1d5d3d480648556d858e2c41c113ed7fdadd973c38a5323125ad73665a04726f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/62327ff9b767ae88ceb96c2c927a501d206124a77f1c58a5baab0b1c19bd05e7/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/2e04427d6180fd6f7a234b787382b7a5238eb3c7f382243e3a3d80cf2d35de63/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/6658220aab6aa710180bdca7f8fa41cffd99cd49f3de1455fd2e4f9c96c795ab/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c07bdc00a8a57f4c3ecda74d7db4a7fdab56a532544db842c313d4c37e10546a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5923a58052d79d822099eb48cebfe5826232676cc4c4a71ed00ab7a5a5f8ad3a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2213e47d46525c698c08e79975b8eb5fda48466d2f9d080babb571552709d1d7/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/dc495f12c2b437ae42959061423e6613a5fa338ad5f33d05d16a984b5244c26c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/77eb1abc055f8b15aed0c6a0e732894fb3042b75513ef697b1642f59c8362f3d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ca518950275ef221b4d8fa52b73b093bb3182c3fc7b7832c852f886f1b686112/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/78534efedaae6d9e87e0050f5a828a500820fc066403ac148f80994fcb08733d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a64db462a3de6ebcee710d004bba4f40e60dd5d305d432a80aad66ee99df4b96/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/6d16e008ea7a1b686c0303d0ac54b9c5d05b6ef80a9f0e26eabd50989a151516/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/c6074055a329674123f657762ef2eb21eb2c99523b12346056dbde5799507cd5/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/a7f077bf241a0853c52875338fcc63ff9e286bcd38e404e7f61498c8511e5989/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/003f2f86668c5067e5aea47e5e5b805f16011b223011ba9a53cdcd0b121cc9fb/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/340d6048771185770cb8550f2ae92fef5d62daeabadee169af6c53f6ffade949/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/df657592ff050644d89bd11ca670ac54fa215a48e83c4cd1bef1d61a965c2545/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8d9a2cd283f7efbe5ec08549588f9b5acfcd0d3492126b9b10da961233be4f7a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/2d31180c2fde63c617936488cedb7ce331c258c2009556387fa8f66573810ea9/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/36f2e5468ca34dbea86c6a31d99b4f9b1d0a9fd2592f4b7bcb649b93e05f7242/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/dd392e0afae7a2b74ca1258f87e74e287acc25e9677bce22dee3cc5d2416b3c8/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d46198cc3bf82322be02e82f50589875332c04a10d3e33371ea6afcd34536607/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/6bc1fdf30062ad1e67acbfdcac4018c22c9e1d9d73b6a535cf9f2d037acc49d6/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/68e8091cf4ccd9fefe5b96ed6614214969c003e224cd6d105c2935ec84227abe/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8346f224cee2cf754f9fe7ba2998cbdaa4dc85aa90c935217ad60a553c444c53/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/c7bdae02cdab52937fa963c00c6b4a694d219606045f124f34a3cb2c3103bc6b/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/4ecde0b6c19a99d2d3043c6f19591535861d6882b2ca0871595e8e7e22d61e3d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/35ff156e09d5211f890213ba97aae492dce639d880dad91724622b9fb595d4bb/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/0baae1a06cbcf32f88a534819705159952fff92babb0b8b02158a417a45ea1f6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/1e33188e05f6a48c17e862e1b62af3cbd6e85121f5916ede28b98714e1bbe106/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/c668ba3f8bc1fd860bd72f32ffee14e65cbf8d45ff1f5e32c8357d9ac59ed455/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e6a0393f6cf4683df0af10ab17619ee42fa0094de6f8c331cd648b3021938436/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/6802d8655326784163410e4626e505256e873ea594c26c982a3f68f81627acfd/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/cd6ea2649a7034c8269a8a4df8f12ce7b446c5e261a7f16965ed6be72bd6502d/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/4795a3871a0cad2449c6a3ea1ef24fce8d05dbbf90479aaf091d05844ce3c3ac/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/391883d1b8e5eae369b7f6f7051227c34bcb760a93a9b9b449c9cf8e4709e681/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/16fb5c538781db335a8689f0f92d82c47b0bd94e03fe14cb40da82d852694c31/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/f54d804e0ae7bc61794fd349585965700fd3fbca0690ad9382f2fb2311b41945/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d3ea75da5955cf9a28c465007861d4378f8dd7acd44bbe905e768c4970d55ff9/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/abaceb2a8d023cc1f4801f0c430726ed6fbd24aefa732f9c66938926754732cd/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/7663c22b3cac49ccc6682406243d9bc9788ec6bdc96a37364f4f0701b50b16f9/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/82ad08b4649d585eadafcbf1c1966b07ed2d50bf95071ea9c6433a7483083916/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/bdf7afa2706acf3a3d4f33121a5719626fc22e6a21c07dc8106ddbe05beea8fa/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/90ea565c44241faa7695413fe5ea3beabc74c6f2355980a9af019f6bb292ae0d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/4ab50215170622e6f977b7951a24d668d0df70d08b3de5b7c9f0e6b4e35f9e70/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/96a053a6f23edd8244361a82f811a94a239574cfba42c2009b255a753e251bbd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/77f1b7dedab5bf0bb4cdc32c53a2b76183d65b5ba0290ad2d6973d3c816519fd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d5ea5752d355747d7d21d753321d9d6d235cf4d86d8b44a4cf6b22de075ac589/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a2d01921e9a66fcd7eb40dcd00946797ceb7c07b86ceb539ba1367d55e31d92a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5c348c45cf427d9ac0f5d2f53fbdfe032f9ab6117779f1ec7b5183fa54f06bfa/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f119f0a78f328b4df8f9ad41ab8930f25af4840f02c19c1f262a453f8dd85d8d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a766608157c7981ca5855b526937779097c0f2761577081fbf059730fb5cc2d4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f12141d95c4662523a5b2d0094dbc92e6a3e41c188eea3128fddce831555ca9b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.568647    5085 manager.go:217] Machine: {Timestamp:2026-10-30 19:24:50.567832526 +0000 UTC m=+0.072236271 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099999 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255c5ea6af2d35681076f152e1ca55 SystemUUID:ec255c5e-a6af-2d35-6810-76f152e1ca55 BootID:0983f3f5-d04b-419c-a8f8-301b85a8c190 Filesystems:[{Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5df2cadb5fc9211de01379287c475e8d1939387e9c36db4ef0539412858b9bb/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d9e730e-03fe-43f4-983e-00a9e6975b6e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~projected/kube-api-access-gd74g DeviceMajor:0 DeviceMinor:361 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~projected/kube-api-access-rn899 DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42ca8ad862f3caedcb0de45ea88bbef7bc20d70d1aa9ef6b773e02c8f77ec633/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~projected/kube-api-access-m5qr2 DeviceMajor:0 DeviceMinor:221 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f/volumes/kubernetes.io~projected/kube-api-access-nc5kc DeviceMajor:0 DeviceMinor:51 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~projected/kube-api-access-8mxzz DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd2969bf894e2b6c74db745b3b67c3444f29cddcf4c6d6c6cccf88aa9f2233f6/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d701203897aa06aca1c35935622a2549e013ef4362472e7af21daba0639a80e/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6516fff1195b37e73e912c4675dd02e21682d11723c9d2432ab1a47e3cc95d8c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f359d55b6d0885dff6677856c9b484872c749797438b4fe42228688b41d9dd0b/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/71223e72-c81e-4dea-ab54-53f1954babe9/volumes/kubernetes.io~projected/kube-api-access-bwq7p DeviceMajor:0 DeviceMinor:346 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3216d44bbd4152661fe5ae5b28bbf4a93d43f341abbc6889fef2bdd838dde34b/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f9c4542-7226-437e-85a9-7d7639ffbab7/volumes/kubernetes.io~projected/kube-api-access-297gw DeviceMajor:0 DeviceMinor:306 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4b032043-f55e-472f-bfae-107a66ab7aa0/volumes/kubernetes.io~projected/kube-api-access-q2bkm DeviceMajor:0 DeviceMinor:351 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81a6af15-f8e9-4b72-b798-ebf0438e83f4/volumes/kubernetes.io~projected/kube-api-access-mm54z DeviceMajor:0 DeviceMinor:240 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45a28670-76ac-4e67-be4b-72e0a64e6766/volumes/kubernetes.io~projected/kube-api-access-2r7cc DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdfd4a6d1fadce0f6f62424be7c5559c87e8bf2998ca2b6bc8be89c1d15baf7c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eebcbf2cc062cae7fd327daa4270dd85b0a239b2dd395e5e242ef6f4cab5c27/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa0d5c4b-3127-404e-9d75-92b008cd639c/volumes/kubernetes.io~projected/kube-api-access-rvdx5 DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe42edc0265546dc1ce57e94010ec89e8d768acc5e630d0bb13bf80d63800da1/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/02ed874a-270d-4a83-b7d3-15d3b10454f2/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~projected/kube-api-access-bnjdt DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~projected/kube-api-access-x9tgj DeviceMajor:0 DeviceMinor:217 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5/volumes/kubernetes.io~projected/kube-api-access-5fzgk DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7/volumes/kubernetes.io~projected/kube-api-access-pmv97 DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6871932b-3edd-47fd-9dcb-bc67f686b852/volumes/kubernetes.io~projected/kube-api-access-jvw87 DeviceMajor:0 DeviceMinor:307 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dcc0e467-8e63-4365-8ef7-d6cc511ca027/volumes/kubernetes.io~projected/kube-api-access-ml52g DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a71c312-1f67-4e38-a2dd-d54417c26e9c/volumes/kubernetes.io~projected/kube-api-access-22mrm DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/520877576fc7a6409f8d4cd8d7cf4dac588436641988e057ab87afd85babb116/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25523fa-8b51-42e3-8e6a-79af2ce1c71c/volumes/kubernetes.io~projected/kube-api-access-b9z8d DeviceMajor:0 DeviceMinor:220 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~projected/kube-api-access-rwbhf DeviceMajor:0 DeviceMinor:236 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0b1c6c7-c184-4847-a6b2-9dc5371171b5/volumes/kubernetes.io~projected/kube-api-access-8bgm7 DeviceMajor:0 DeviceMinor:54 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/189d14bfd06f3c15affe7b9c62bad9b32552251c71f82a07791ad79c32f1b508/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dcdfa192d050b0abe563aecdfcd1f7b4d894ba73ed6c1e7044aa3fd34fdc39c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/68246332-14ef-4b27-b072-c6141c820f53/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e4802df7bd65e92aeb7c3576f4f151832c8925431232efe8a784b137b70107a/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cd508dfd709efd272e8f94dc2bb664ad092f803c6023c0a29a6f69b84b66608/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c1f9acdb-b018-4e51-aab4-27a6362dc070/volumes/kubernetes.io~projected/kube-api-access-f2vhh DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volumes/kubernetes.io~projected/kube-api-access-pn2m9 DeviceMajor:0 DeviceMinor:53 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/95cc74a0-83aa-4d4b-9fa9-b604093c4e97/volumes/kubernetes.io~projected/kube-api-access-4rglf DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7785b6b25065f5e28281ee4544465776901352c8a2640eb10fd6848ddcf9748/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0daabe8-7547-4768-9157-55e9a70f1fc1/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/11477705-d48e-4ba3-a15a-8aba013e83cc/volumes/kubernetes.io~projected/kube-api-access-rtdrq DeviceMajor:0 DeviceMinor:215 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8c679173-6012-49d3-9a8b-f1b846552a35/volumes/kubernetes.io~projected/kube-api-access-njcmd DeviceMajor:0 DeviceMinor:210 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~projected/kube-api-access-7w26d DeviceMajor:0 DeviceMinor:216 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e446f841a735c128e9f68388803f7ede348d2657cc5ca2bc977250adc5a720d3/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8944ed8b-8a0f-485b-88e4-177f50a9f437/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4307a39472814865fdb8c4496b380c7a9405c799d6b6ca1434902b0c129a99ae/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:208 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d690fb63-2b99-4a20-8a76-4650ce687a0f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/983dc1c4-4111-46f2-a526-f712eb5e56bf/volumes/kubernetes.io~projected/kube-api-access-hfmls DeviceMajor:0 DeviceMinor:52 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f913148-3853-4762-a98a-148e2fdb44b8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/925c520cc377ed382c94567c7c72dd07ba6580049e3a71cc6945223a17fd8bca/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/322552eb-09e0-405f-b174-a7e3317605d3/volumes/kubernetes.io~projected/kube-api-access-mw2sm DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~projected/kube-api-access-mxmfv DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e407fc34-c94f-4bfe-adf1-5ec24e48de7e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a6b1d077d9fc6afddc362d2c51dfdbe889655e30680525926a8a62ea8623418/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a6253bc917ef66999c15c2010548d637674afe7ae76d4cc21490e7399ee3047/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90cd1cc582c32ec45d399a72882fcfcb37e1b6097b182fe6770d223c5480e290/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da0f2ad5b6796ed85c10b933f0ff164f65599def10ceaf3f00ea8f8d76df69b5/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ecfafb0f-0c2e-4703-be40-c9c2a080fd48/volumes/kubernetes.io~projected/kube-api-access-wr6gk DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:189d14bfd06f3c1 MacAddress:96:d7:b1:2c:3e:5c Speed:10000 Mtu:8901} {Name:2a6b1d077d9fc6a MacAddress:d6:c2:ca:ee:64:eb Speed:10000 Mtu:8901} {Name:3216d44bbd41526 MacAddress:c6:2a:d3:e3:a5:50 Speed:10000 Mtu:8901} {Name:3d701203897aa06 MacAddress:a6:cc:e0:39:f6:52 Speed:10000 Mtu:8901} {Name:42ca8ad862f3cae MacAddress:1a:e0:01:be:17:1e Speed:10000 Mtu:8901} {Name:4307a3947281486 MacAddress:32:3a:36:dc:1e:9d Speed:10000 Mtu:8901} {Name:7e4802df7bd65e9 MacAddress:92:c7:39:0e:38:18 Speed:10000 Mtu:8901} {Name:8cd508dfd709efd MacAddress:be:52:6f:da:34:54 Speed:10000 Mtu:8901} {Name:925c520cc377ed3 MacAddress:42:32:7e:5f:62:61 Speed:10000 Mtu:8901} {Name:b2ddb2c47a8c6d7 MacAddress:fe:a6:58:e3:57:cb Speed:10000 Mtu:8901} {Name:bdfd4a6d1fadce0 MacAddress:2a:d0:e1:f4:f5:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c7:76:0d:1f Speed:0 Mtu:9001} {Name:br-int MacAddress:62:3d:63:d8:67:32 Speed:0 Mtu:8901} {Name:cd2969bf894e2b6 MacAddress:86:19:96:46:20:a7 Speed:10000 Mtu:8901} {Name:e446f841a735c12 MacAddress:f2:f0:28:b3:a7:20 Speed:10000 Mtu:8901} {Name:e5df2cadb5fc921 MacAddress:6a:13:7a:df:a2:c5 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c7:76:0d:1f Speed:0 Mtu:9001} {Name:f359d55b6d0885d MacAddress:be:f6:2d:86:5e:96 Speed:10000 Mtu:8901} {Name:fe42edc0265546d MacAddress:ce:03:d7:da:50:f8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:2e:1a:77:a2:5f:20 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:ad:73:5d:d2:ad Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569302    5085 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569379    5085 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569678    5085 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569699    5085 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-200.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569829    5085 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569837    5085 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569847    5085 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569866    5085 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.569885    5085 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570030    5085 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570122    5085 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570154    5085 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570173    5085 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570183    5085 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570192    5085 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:24:50.570476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.570201    5085 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:24:50.571148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.571041    5085 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:24:50.571148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.571051    5085 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:24:50.575340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.575319    5085 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:24:50.578497 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578482    5085 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:24:50.578537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578497    5085 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:24:50.578628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578617    5085 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:24:50.578827 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578816    5085 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:24:50.578858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578845    5085 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:24:50.578949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578940    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:24:50.578981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578954    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:24:50.578981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578960    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:24:50.578981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578966    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:24:50.578981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578972    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:24:50.578981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578978    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578987    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.578996    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579004    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579010    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579024    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579033    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579057    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:24:50.579109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.579066    5085 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:24:50.582495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.582482    5085 server.go:1263] "Started kubelet"
Oct 30 19:24:50.582627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.582593    5085 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:24:50.582682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.582597    5085 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:24:50.582745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.582724    5085 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:24:50.583139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.583120    5085 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:24:50.583533 ip-10-0-134-200 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:24:50.585995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.584854    5085 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.588110    5085 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.588222    5085 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.588231    5085 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.588246    5085 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.588414    5085 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.589074    5085 factory.go:55] Registering systemd factory
Oct 30 19:24:50.589688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.589092    5085 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.589988    5085 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590052    5085 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:19:07 +0000 UTC" deadline="2028-06-05 05:07:30.848159022 +0000 UTC"
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590088    5085 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14001h42m40.258075113s"
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590489    5085 factory.go:153] Registering CRI-O factory
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590521    5085 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590599    5085 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590619    5085 factory.go:103] Registering Raw factory
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590635    5085 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:24:50.591711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.590984    5085 manager.go:319] Starting recovery of all containers
Oct 30 19:24:50.596368 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596331    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a71c312-1f67-4e38-a2dd-d54417c26e9c" volumeName="kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596370    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596386    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51992eb3-1df9-4e7f-9541-a9e5c8aab9f7" volumeName="kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596398    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6871932b-3edd-47fd-9dcb-bc67f686b852" volumeName="kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596413    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25523fa-8b51-42e3-8e6a-79af2ce1c71c" volumeName="kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596426    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" volumeName="kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:24:50.596441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596439    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa0d5c4b-3127-404e-9d75-92b008cd639c" volumeName="kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596451    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596473    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11477705-d48e-4ba3-a15a-8aba013e83cc" volumeName="kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596485    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e59d18a-360e-4921-a459-37f3557f9012" volumeName="kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596497    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6871932b-3edd-47fd-9dcb-bc67f686b852" volumeName="kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596509    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95cc74a0-83aa-4d4b-9fa9-b604093c4e97" volumeName="kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596530    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407fc34-c94f-4bfe-adf1-5ec24e48de7e" volumeName="kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596547    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596579    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596592    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596604    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ed874a-270d-4a83-b7d3-15d3b10454f2" volumeName="kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned" seLinuxMountContext=""
Oct 30 19:24:50.596630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596626    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ed874a-270d-4a83-b7d3-15d3b10454f2" volumeName="kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596638    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596650    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f9c4542-7226-437e-85a9-7d7639ffbab7" volumeName="kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596662    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596673    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322552eb-09e0-405f-b174-a7e3317605d3" volumeName="kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596686    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c679173-6012-49d3-9a8b-f1b846552a35" volumeName="kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596698    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0daabe8-7547-4768-9157-55e9a70f1fc1" volumeName="kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596714    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45a28670-76ac-4e67-be4b-72e0a64e6766" volumeName="kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596726    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0daabe8-7547-4768-9157-55e9a70f1fc1" volumeName="kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596737    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83467d2-3710-458a-8ec7-a8cc4d2c4b3f" volumeName="kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596748    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc0e467-8e63-4365-8ef7-d6cc511ca027" volumeName="kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596765    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407fc34-c94f-4bfe-adf1-5ec24e48de7e" volumeName="kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596776    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596786    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68246332-14ef-4b27-b072-c6141c820f53" volumeName="kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596799    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25523fa-8b51-42e3-8e6a-79af2ce1c71c" volumeName="kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596811    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" volumeName="kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596822    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d690fb63-2b99-4a20-8a76-4650ce687a0f" volumeName="kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596833    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8944ed8b-8a0f-485b-88e4-177f50a9f437" volumeName="kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596845    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407fc34-c94f-4bfe-adf1-5ec24e48de7e" volumeName="kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596857    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6871932b-3edd-47fd-9dcb-bc67f686b852" volumeName="kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596868    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983dc1c4-4111-46f2-a526-f712eb5e56bf" volumeName="kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596880    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" volumeName="kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596891    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfafb0f-0c2e-4703-be40-c9c2a080fd48" volumeName="kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596903    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f259bbb5-00a3-4a12-92e5-7ea6e2801ec5" volumeName="kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk" seLinuxMountContext=""
Oct 30 19:24:50.596909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596915    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b032043-f55e-472f-bfae-107a66ab7aa0" volumeName="kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596927    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95cc74a0-83aa-4d4b-9fa9-b604093c4e97" volumeName="kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596938    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983dc1c4-4111-46f2-a526-f712eb5e56bf" volumeName="kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596950    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b9a83a-3e00-46aa-9cdd-4508a3f917eb" volumeName="kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596962    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f" volumeName="kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596975    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596987    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407fc34-c94f-4bfe-adf1-5ec24e48de7e" volumeName="kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.596998    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0daabe8-7547-4768-9157-55e9a70f1fc1" volumeName="kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597011    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597022    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="322552eb-09e0-405f-b174-a7e3317605d3" volumeName="kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597034    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68246332-14ef-4b27-b072-c6141c820f53" volumeName="kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597046    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6af15-f8e9-4b72-b798-ebf0438e83f4" volumeName="kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597057    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b9a83a-3e00-46aa-9cdd-4508a3f917eb" volumeName="kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597068    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfafb0f-0c2e-4703-be40-c9c2a080fd48" volumeName="kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597080    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597099    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51992eb3-1df9-4e7f-9541-a9e5c8aab9f7" volumeName="kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597111    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6af15-f8e9-4b72-b798-ebf0438e83f4" volumeName="kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597125    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" volumeName="kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597137    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc0e467-8e63-4365-8ef7-d6cc511ca027" volumeName="kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597151    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d9e730e-03fe-43f4-983e-00a9e6975b6e" volumeName="kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597163    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1f9acdb-b018-4e51-aab4-27a6362dc070" volumeName="kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597175    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ed874a-270d-4a83-b7d3-15d3b10454f2" volumeName="kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597186    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597199    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597213    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f" volumeName="kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597226    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597239    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597251    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597265    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597278    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0daabe8-7547-4768-9157-55e9a70f1fc1" volumeName="kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597292    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d690fb63-2b99-4a20-8a76-4650ce687a0f" volumeName="kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597304    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c179c5b-fe07-4373-889d-2807b0e0a206" volumeName="kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597317    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f9c4542-7226-437e-85a9-7d7639ffbab7" volumeName="kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597328    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f913148-3853-4762-a98a-148e2fdb44b8" volumeName="kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597340    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68246332-14ef-4b27-b072-c6141c820f53" volumeName="kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597352    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71223e72-c81e-4dea-ab54-53f1954babe9" volumeName="kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597364    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81a6af15-f8e9-4b72-b798-ebf0438e83f4" volumeName="kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597376    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8944ed8b-8a0f-485b-88e4-177f50a9f437" volumeName="kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597389    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9b9a83a-3e00-46aa-9cdd-4508a3f917eb" volumeName="kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597416    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6871932b-3edd-47fd-9dcb-bc67f686b852" volumeName="kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597430    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc0e467-8e63-4365-8ef7-d6cc511ca027" volumeName="kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597441    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f" volumeName="kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597453    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e407fc34-c94f-4bfe-adf1-5ec24e48de7e" volumeName="kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597463    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecfafb0f-0c2e-4703-be40-c9c2a080fd48" volumeName="kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597471    5085 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fff74ec7-d916-4f36-a3e8-a77eb7649121" volumeName="kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets" seLinuxMountContext=""
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597479    5085 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:24:50.597503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.597484    5085 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:24:50.610450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.610426    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:50.610966 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.610948    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:50.621511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.621476    5085 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:24:50.623129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.623106    5085 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:24:50.623229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.623135    5085 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:24:50.623229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.623164    5085 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:24:50.623229 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.623199    5085 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:24:50.650163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.650106    5085 generic.go:356] "Generic (PLEG): container finished" podID="16650c5b012d27eb875de166ae5ed482" containerID="5bb7a40732fff2a59bf9001e52177054c399099559fbe16ed44a43e98b68e2f2" exitCode=0
Oct 30 19:24:50.668762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.668697    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:50.668762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.668724    5085 generic.go:356] "Generic (PLEG): container finished" podID="b0daabe8-7547-4768-9157-55e9a70f1fc1" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb" exitCode=255
Oct 30 19:24:50.679941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679697    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="88b980d48f605e65446da3bca85f8329592d59f328406f2cc09e05d12f14b90d" exitCode=0
Oct 30 19:24:50.679941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679942    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="a699990ddce3c38a357e26141ef7959b1cfddb11eebf01a0fe92deaef1365e41" exitCode=0
Oct 30 19:24:50.680105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679955    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="9d5b98b42a65fa448b2f5f759e3ce422e44d00e3e29985b7bd24df6b058eee5f" exitCode=0
Oct 30 19:24:50.680105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679965    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="3b22b73fa4a4944d138a4ed2364df0beceb80107a9951996de3ee48ec8c7e117" exitCode=0
Oct 30 19:24:50.680105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679974    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="fa06c3dc79ae2d3970289764046a627b329dc95800838f01863557ced13df9a9" exitCode=0
Oct 30 19:24:50.680105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.679983    5085 generic.go:356] "Generic (PLEG): container finished" podID="d0b1c6c7-c184-4847-a6b2-9dc5371171b5" containerID="7617149aa095dff42b45f37994798cc6df5a336943a961b6923706eb7a0935e8" exitCode=0
Oct 30 19:24:50.688312 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.688292    5085 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:50.693370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.693352    5085 manager.go:324] Recovery completed
Oct 30 19:24:50.710829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.710808    5085 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:50.711930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.711916    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:50.711988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.711944    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:50.711988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.711954    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:50.714898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.714878    5085 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:24:50.714898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.714896    5085 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:24:50.715071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.714918    5085 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:24:50.715159 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715142    5085 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:24:50.715229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715161    5085 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:24:50.715229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715217    5085 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:24:50.715229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715226    5085 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:24:50.715375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715235    5085 policy_none.go:50] "Start"
Oct 30 19:24:50.715375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715255    5085 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:24:50.715375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715271    5085 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:24:50.715520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715510    5085 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:24:50.715604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715523    5085 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:24:50.715604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715532    5085 policy_none.go:44] "Start"
Oct 30 19:24:50.715705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.715655    5085 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:50.716828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.716808    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:50.716915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.716842    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:50.716915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.716853    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:50.720395 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.720361    5085 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:24:50.723936 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.723917    5085 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 19:24:50.725742 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.725729    5085 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:24:50.725812 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.725753    5085 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:24:50.725812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.725762    5085 server.go:87] "Starting device plugin registration server"
Oct 30 19:24:50.725945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.725933    5085 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:24:50.725996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.725946    5085 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:24:50.726062 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.726046    5085 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:24:50.726174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.726140    5085 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:24:50.726228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.726184    5085 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:24:50.726228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.726191    5085 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:24:50.726373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.726275    5085 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:24:50.726173298Z","UUID":"1e2c6034-6008-41c1-8685-f008c0b3b6c8","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:24:50.727724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.727702    5085 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:24:50.727816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.727751    5085 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:24:50.727816 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.727795    5085 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:24:50.727897 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.727844    5085 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-200.us-west-2.compute.internal\" not found"
Oct 30 19:24:50.778739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.778711    5085 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:50.778739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.778719    5085 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:50.778913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.778776    5085 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:24:50.823993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.823965    5085 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:50.826025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.826008    5085 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:24:50.826470 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.826452    5085 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:24:50.827159 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.827136    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:24:50.827262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.827172    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:24:50.827262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.827187    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:24:50.827367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.827275    5085 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.924726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.924669    5085 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"]
Oct 30 19:24:50.925439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925393    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"01a2322c0b6f8492dadef7e7f75813b1","Type":"ContainerStarted","Data":"4b82182c5b0b339c43d8c383cdc888d55d1358f48ed9fca59c5e6251e13d3927"}
Oct 30 19:24:50.925552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925454    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"01a2322c0b6f8492dadef7e7f75813b1","Type":"ContainerStarted","Data":"f45b63ad2581c33e2414793d7ec530feca571ad9258aeec256caff374e73d8ed"}
Oct 30 19:24:50.925552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925539    5085 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.925685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925590    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerStarted","Data":"1b8a2bb26fb76213359bbf98b798db5092cbdb59772579a2e9aaacb3ca1b88ab"}
Oct 30 19:24:50.925685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925634    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerDied","Data":"5bb7a40732fff2a59bf9001e52177054c399099559fbe16ed44a43e98b68e2f2"}
Oct 30 19:24:50.925685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925655    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal" event={"ID":"16650c5b012d27eb875de166ae5ed482","Type":"ContainerStarted","Data":"01d7dce93e82424ad537cd039f22e999c8137dbcf89467e622d13de2795bf168"}
Oct 30 19:24:50.925819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.925688    5085 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.932055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.932036    5085 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.932124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.932090    5085 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.932644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.932625    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 19:24:50.932700 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.932652    5085 setters.go:546] "Node became not ready" node="ip-10-0-134-200.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:24:50Z","lastTransitionTime":"2026-10-30T19:24:50Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 19:24:50.934049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.934032    5085 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-200.us-west-2.compute.internal" is owned by different node (oldNodeID="3e9f5a70-7b31-4feb-965e-8a8700005731", newNodeID=""), cleaning up...
Oct 30 19:24:50.987961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.987931    5085 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:50.988110 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.987979    5085 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:50.988876 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:50.988861    5085 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:50.988939 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:50.988896    5085 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.080279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.080239    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.080279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.080276    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.080471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.080295    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181033    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181070    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181092    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181364 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181138    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181364 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181142    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/16650c5b012d27eb875de166ae5ed482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"16650c5b012d27eb875de166ae5ed482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.181364 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.181192    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/01a2322c0b6f8492dadef7e7f75813b1-config\") pod \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" (UID: \"01a2322c0b6f8492dadef7e7f75813b1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.570398 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.570316    5085 apiserver.go:52] "Watching apiserver"
Oct 30 19:24:51.687807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.687773    5085 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.687969 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.687888    5085 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.714829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.714803    5085 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:24:51.715513 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.715478    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtrx2","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9","openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b","openshift-image-registry/node-ca-4pq5w","openshift-insights/periodic-gathering-2md8m-pt29t","openshift-network-console/networking-console-plugin-86c478bb88-sltb5","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8","openshift-dns/node-resolver-gxfnz","openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc","openshift-network-operator/iptables-alerter-xs627","openshift-service-ca/service-ca-5fffc97bdf-nsjps","openshift-cluster-node-tuning-operator/tuned-g59hg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal","kube-system/global-pull-secret-syncer-vg85t","openshift-dns/dns-default-54xrz","openshift-insights/insights-runtime-extractor-qbzm8","openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st","openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw","kube-system/kas-connection-checker-685895c7c9-ldqhw","openshift-ovn-kubernetes/ovnkube-node-rx5vx","openshift-console-operator/console-operator-7cd56b8dbc-gc2t4","openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj","kube-system/kas-connection-checker-685895c7c9-xx99q","kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-q6ssb","kube-system/konnectivity-agent-qhxdt","openshift-network-diagnostics/network-check-target-cjw9x","openshift-multus/multus-w8279","openshift-multus/network-metrics-daemon-lf6jh","openshift-ingress/router-default-8598d5bb4c-4xl9z","openshift-image-registry/image-registry-7c5856f576-f75k6","openshift-ingress-canary/ingress-canary-sh5vv","openshift-insights/insights-operator-7c4986d449-zdmh7","openshift-image-registry/image-registry-f75bb8685-6qjls","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"]
Oct 30 19:24:51.717379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.717358    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.717778 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.717758    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.719826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.719807    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.720282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.720267    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.720756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.720739    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:51.728556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.728538    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:51.730198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.730181    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.731437 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.731421    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.732051 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.732036    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.743894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.743863    5085 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:51.743980 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.743936    5085 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.744697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.744683    5085 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:24:51.744742 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.744720    5085 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-200.us-west-2.compute.internal"
Oct 30 19:24:51.765607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.765589    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.766005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.765981    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.766005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.765981    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.766155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766045    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.766155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766111    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.766252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766166    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.766252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766176    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 19:24:51.766684 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766671    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.766929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.766915    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.767835 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.767818    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.768275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768113    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 19:24:51.768275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768131    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.768275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768209    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.768523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768297    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 19:24:51.768863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768840    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qhxtv\""
Oct 30 19:24:51.768863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.768856    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5wmd9\""
Oct 30 19:24:51.769135 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769117    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 19:24:51.769305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769291    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 19:24:51.769541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769528    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.769910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769893    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.769910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769899    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.770047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769948    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 19:24:51.770047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769955    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 19:24:51.770047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769960    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 19:24:51.770047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.769966    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 19:24:51.770199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.770059    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.792012 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.791992    5085 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:24:51.811833 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.811812    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 19:24:51.812606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.812294    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 19:24:51.812606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.812419    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 19:24:51.812606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.812446    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-4qk4l\""
Oct 30 19:24:51.812606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.812468    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 19:24:51.812606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.812545    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-w6qzk\""
Oct 30 19:24:51.813112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.813088    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 19:24:51.813446 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.813423    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zgmvh\""
Oct 30 19:24:51.813495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.813459    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-llrr6\""
Oct 30 19:24:51.817808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.817784    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:24:51.818304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818284    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 19:24:51.818766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818748    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-t2d46\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818779    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818803    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818813    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818845    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hthrr\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818857    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7vvd5\""
Oct 30 19:24:51.818871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818841    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818899    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818923    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818943    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.818983    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819002    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819010    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819074    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819081    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819078    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.819125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819126    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819160    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819178    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819197    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819212    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819232    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-s4fj9\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819254    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819275    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vwkbz\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819332    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819351    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819388    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819392    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819402    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jxzqp\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819394    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819411    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819443    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819436    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819461    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819467    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819474    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819515    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819529    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lzgjh\""
Oct 30 19:24:51.819539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819535    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819610    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-46rfm\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819694    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819728    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6ftg5\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819738    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-58c9c\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819800    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819840    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6prqp\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819880    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819882    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819937    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zkvzq\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.819995    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6nsf6\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820009    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wzkg2\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820379    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820383    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820379    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820436    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820474    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820519    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:24:51.820547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820530    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820592    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820619    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820644    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820661    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820707    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820835    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8x78l\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820908    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820952    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-j9w96\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.820975    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9jbqb\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821004    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821020    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f29wx\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821046    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821083    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ndzf5\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821113    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821147    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821185    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-t58hx\""
Oct 30 19:24:51.821342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821288    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:24:51.822076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821370    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4g88n\""
Oct 30 19:24:51.822076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821447    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9th84\""
Oct 30 19:24:51.822076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.821601    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:24:51.886040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886010    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.886171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886044    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.886171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886064    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.886171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886082    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:51.886171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886137    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.886171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886166    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.886332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886186    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.886332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886209    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.886332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886257    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.886332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886281    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.886473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886328    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.886531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886506    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1e59d18a-360e-4921-a459-37f3557f9012-nginx-conf\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:51.886600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886581    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.886643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886599    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-script-lib\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.886643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886619    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.886722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886646    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.886767 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886708    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.886767 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886748    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.886919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886773    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-service-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.886919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886815    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-key\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.886919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886839    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.886919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886887    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd74g\" (UniqueName: \"kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886946    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886974    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.886984    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z8d\" (UniqueName: \"kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887013    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6gk\" (UniqueName: \"kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887020    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovnkube-config\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.887093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887038    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.887325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887190    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-stats-auth\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.887325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887193    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.887325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887228    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbhf\" (UniqueName: \"kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:51.887325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887295    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmfv\" (UniqueName: \"kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.887485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887348    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.887485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887381    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.887485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887420    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:51.887485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887452    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.887485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887468    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwq7p\" (UniqueName: \"kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p\") pod \"migrator-66df5d496b-n78wc\" (UID: \"71223e72-c81e-4dea-ab54-53f1954babe9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887485    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887507    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887581    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f9c4542-7226-437e-85a9-7d7639ffbab7-data-volume\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887550    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887622    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2m9\" (UniqueName: \"kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887646    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.887731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887696    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887735    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887783    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887789    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887798    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887827    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887885    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887923    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.887965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887949    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.887975    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888010    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888030    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888043    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888123    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888151    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888176    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888176    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888194    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888213    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888231    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.888283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888247    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888290    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888347    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888389    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888444    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888451    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cni-binary-copy\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888473    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297gw\" (UniqueName: \"kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888533    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81a6af15-f8e9-4b72-b798-ebf0438e83f4-tmp-dir\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888556    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888604    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888634    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm54z\" (UniqueName: \"kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888661    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888687    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888697    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-snapshots\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888714    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888764    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888826    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888842    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.888894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888858    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888918    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888940    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-serviceca\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.888992    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8944ed8b-8a0f-485b-88e4-177f50a9f437-cert\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889020    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889068    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmls\" (UniqueName: \"kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889072    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-daemon-config\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889087    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d9e730e-03fe-43f4-983e-00a9e6975b6e-original-pull-secret\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889109    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzgk\" (UniqueName: \"kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk\") pod \"network-check-source-6f78d59f99-lskdw\" (UID: \"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889172    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw2sm\" (UniqueName: \"kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889254    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5kc\" (UniqueName: \"kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889280    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889293    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d690fb63-2b99-4a20-8a76-4650ce687a0f-agent-certs\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889306    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml52g\" (UniqueName: \"kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889332    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxzz\" (UniqueName: \"kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889357    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889366    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-signing-cabundle\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889381    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889409    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889442    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889470    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.889531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889506    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889542    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889625    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889681    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889710    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889737    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889755    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889759    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889801    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjdt\" (UniqueName: \"kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889812    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f913148-3853-4762-a98a-148e2fdb44b8-tmp\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889807    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-config\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889834    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889850    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889869    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9tgj\" (UniqueName: \"kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889894    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.889928    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890005    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.890096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890046    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890156    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890238    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890282    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890340    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81a6af15-f8e9-4b72-b798-ebf0438e83f4-config-volume\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890374    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmv97\" (UniqueName: \"kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890404    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890418    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/68246332-14ef-4b27-b072-c6141c820f53-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890432    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890456    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/68246332-14ef-4b27-b072-c6141c820f53-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890534    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890614    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.890683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890647    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdx5\" (UniqueName: \"kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5\") pod \"volume-data-source-validator-58c9f5f46d-fc8s9\" (UID: \"fa0d5c4b-3127-404e-9d75-92b008cd639c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890738    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890740    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-trusted-ca\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890790    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890829    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890845    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890872    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890887    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890901    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.890917    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.891109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891043    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b0daabe8-7547-4768-9157-55e9a70f1fc1-config\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.891419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891259    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-serving-cert\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.891419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891279    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b0daabe8-7547-4768-9157-55e9a70f1fc1-serving-cert\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.891419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891322    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.891419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891360    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-env-overrides\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.891419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891364    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.891552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891472    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.891552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891496    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.891552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891505    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bkm\" (UniqueName: \"kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm\") pod \"migrator-66df5d496b-7b6st\" (UID: \"4b032043-f55e-472f-bfae-107a66ab7aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:51.891552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891536    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:51.891688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891603    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.891688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891633    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.891688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891653    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcmd\" (UniqueName: \"kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd\") pod \"kas-connection-checker-685895c7c9-ldqhw\" (UID: \"8c679173-6012-49d3-9a8b-f1b846552a35\") " pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:51.891827 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891815    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:51.891860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891841    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.891923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891908    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.891967 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891946    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-tuned\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.891999 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891980    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r7cc\" (UniqueName: \"kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc\") pod \"kas-connection-checker-685895c7c9-rtrx2\" (UID: \"45a28670-76ac-4e67-be4b-72e0a64e6766\") " pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:51.891999 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.891991    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d690fb63-2b99-4a20-8a76-4650ce687a0f-konnectivity-ca\") pod \"konnectivity-agent-qhxdt\" (UID: \"d690fb63-2b99-4a20-8a76-4650ce687a0f\") " pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:24:51.892055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892011    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.892055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892015    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02ed874a-270d-4a83-b7d3-15d3b10454f2-tmp\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.892108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892100    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.892140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892117    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.892140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892134    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.892200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892165    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.892200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892176    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.892271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892252    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.892326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892310    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-default-certificate\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.892355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892349    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.892389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892374    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.892419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892403    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.892450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892416    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-ovn-node-metrics-cert\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.892450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892437    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rglf\" (UniqueName: \"kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.892509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892466    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22mrm\" (UniqueName: \"kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:51.892509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892493    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.892587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.892587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892549    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.892587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892577    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.892708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892594    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.892708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892645    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgm7\" (UniqueName: \"kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.892708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892677    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.892847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892713    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.892847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892745    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.892847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892774    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.892847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892781    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/983dc1c4-4111-46f2-a526-f712eb5e56bf-tmp-dir\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.892847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892809    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892850    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892875    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892944    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtdrq\" (UniqueName: \"kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq\") pod \"kas-connection-checker-685895c7c9-xx99q\" (UID: \"11477705-d48e-4ba3-a15a-8aba013e83cc\") " pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892953    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2f913148-3853-4762-a98a-148e2fdb44b8-serving-cert\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.892972    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893000    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893031    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/322552eb-09e0-405f-b174-a7e3317605d3-iptables-alerter-script\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893034    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893032    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.893060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893063    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893088    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn899\" (UniqueName: \"kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893092    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f913148-3853-4762-a98a-148e2fdb44b8-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893141    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893270    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893285    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.893402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.893299    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.945907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.945875    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwq7p\" (UniqueName: \"kubernetes.io/projected/71223e72-c81e-4dea-ab54-53f1954babe9-kube-api-access-bwq7p\") pod \"migrator-66df5d496b-n78wc\" (UID: \"71223e72-c81e-4dea-ab54-53f1954babe9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-n78wc"
Oct 30 19:24:51.945907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.945896    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6gk\" (UniqueName: \"kubernetes.io/projected/ecfafb0f-0c2e-4703-be40-c9c2a080fd48-kube-api-access-wr6gk\") pod \"service-ca-operator-544d749c8c-xv9tj\" (UID: \"ecfafb0f-0c2e-4703-be40-c9c2a080fd48\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj"
Oct 30 19:24:51.946097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.946081    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vhh\" (UniqueName: \"kubernetes.io/projected/c1f9acdb-b018-4e51-aab4-27a6362dc070-kube-api-access-f2vhh\") pod \"network-check-target-cjw9x\" (UID: \"c1f9acdb-b018-4e51-aab4-27a6362dc070\") " pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:51.946148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.946133    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmfv\" (UniqueName: \"kubernetes.io/projected/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-kube-api-access-mxmfv\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.946180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.946131    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z8d\" (UniqueName: \"kubernetes.io/projected/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-kube-api-access-b9z8d\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.946180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.946131    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbhf\" (UniqueName: \"kubernetes.io/projected/8944ed8b-8a0f-485b-88e4-177f50a9f437-kube-api-access-rwbhf\") pod \"ingress-canary-sh5vv\" (UID: \"8944ed8b-8a0f-485b-88e4-177f50a9f437\") " pod="openshift-ingress-canary/ingress-canary-sh5vv"
Oct 30 19:24:51.947055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.947034    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm54z\" (UniqueName: \"kubernetes.io/projected/81a6af15-f8e9-4b72-b798-ebf0438e83f4-kube-api-access-mm54z\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.947254 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.947237    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd74g\" (UniqueName: \"kubernetes.io/projected/e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f-kube-api-access-gd74g\") pod \"service-ca-5fffc97bdf-nsjps\" (UID: \"e2a88398-db0d-46cd-8cbb-ac92a2f2eb9f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nsjps"
Oct 30 19:24:51.947657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.947639    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmv97\" (UniqueName: \"kubernetes.io/projected/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-kube-api-access-pmv97\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.947897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.947877    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmls\" (UniqueName: \"kubernetes.io/projected/983dc1c4-4111-46f2-a526-f712eb5e56bf-kube-api-access-hfmls\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.948018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.948001    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.948139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.948123    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw2sm\" (UniqueName: \"kubernetes.io/projected/322552eb-09e0-405f-b174-a7e3317605d3-kube-api-access-mw2sm\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.948348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.948331    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2m9\" (UniqueName: \"kubernetes.io/projected/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-kube-api-access-pn2m9\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.949143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.949127    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxzz\" (UniqueName: \"kubernetes.io/projected/02ed874a-270d-4a83-b7d3-15d3b10454f2-kube-api-access-8mxzz\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.950340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.950321    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.950704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.950688    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9tgj\" (UniqueName: \"kubernetes.io/projected/2f913148-3853-4762-a98a-148e2fdb44b8-kube-api-access-x9tgj\") pod \"insights-operator-7c4986d449-zdmh7\" (UID: \"2f913148-3853-4762-a98a-148e2fdb44b8\") " pod="openshift-insights/insights-operator-7c4986d449-zdmh7"
Oct 30 19:24:51.950745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.950728    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.953096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953071    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtdrq\" (UniqueName: \"kubernetes.io/projected/11477705-d48e-4ba3-a15a-8aba013e83cc-kube-api-access-rtdrq\") pod \"kas-connection-checker-685895c7c9-xx99q\" (UID: \"11477705-d48e-4ba3-a15a-8aba013e83cc\") " pod="kube-system/kas-connection-checker-685895c7c9-xx99q"
Oct 30 19:24:51.953162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953111    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r7cc\" (UniqueName: \"kubernetes.io/projected/45a28670-76ac-4e67-be4b-72e0a64e6766-kube-api-access-2r7cc\") pod \"kas-connection-checker-685895c7c9-rtrx2\" (UID: \"45a28670-76ac-4e67-be4b-72e0a64e6766\") " pod="kube-system/kas-connection-checker-685895c7c9-rtrx2"
Oct 30 19:24:51.953204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953164    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjdt\" (UniqueName: \"kubernetes.io/projected/68246332-14ef-4b27-b072-c6141c820f53-kube-api-access-bnjdt\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8\" (UID: \"68246332-14ef-4b27-b072-c6141c820f53\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8"
Oct 30 19:24:51.953412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953396    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgm7\" (UniqueName: \"kubernetes.io/projected/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-kube-api-access-8bgm7\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.953500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953472    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdx5\" (UniqueName: \"kubernetes.io/projected/fa0d5c4b-3127-404e-9d75-92b008cd639c-kube-api-access-rvdx5\") pod \"volume-data-source-validator-58c9f5f46d-fc8s9\" (UID: \"fa0d5c4b-3127-404e-9d75-92b008cd639c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fc8s9"
Oct 30 19:24:51.953662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.953645    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzgk\" (UniqueName: \"kubernetes.io/projected/f259bbb5-00a3-4a12-92e5-7ea6e2801ec5-kube-api-access-5fzgk\") pod \"network-check-source-6f78d59f99-lskdw\" (UID: \"f259bbb5-00a3-4a12-92e5-7ea6e2801ec5\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-lskdw"
Oct 30 19:24:51.954167 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.954148    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22mrm\" (UniqueName: \"kubernetes.io/projected/1a71c312-1f67-4e38-a2dd-d54417c26e9c-kube-api-access-22mrm\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:51.954388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.954372    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml52g\" (UniqueName: \"kubernetes.io/projected/dcc0e467-8e63-4365-8ef7-d6cc511ca027-kube-api-access-ml52g\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.954535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.954521    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.955737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.955718    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcmd\" (UniqueName: \"kubernetes.io/projected/8c679173-6012-49d3-9a8b-f1b846552a35-kube-api-access-njcmd\") pod \"kas-connection-checker-685895c7c9-ldqhw\" (UID: \"8c679173-6012-49d3-9a8b-f1b846552a35\") " pod="kube-system/kas-connection-checker-685895c7c9-ldqhw"
Oct 30 19:24:51.955955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.955938    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"periodic-gathering-2md8m-pt29t\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") " pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:24:51.956376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.956356    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5kc\" (UniqueName: \"kubernetes.io/projected/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kube-api-access-nc5kc\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.957460 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.957442    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297gw\" (UniqueName: \"kubernetes.io/projected/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-api-access-297gw\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.958009 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.957989    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rglf\" (UniqueName: \"kubernetes.io/projected/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-kube-api-access-4rglf\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.958504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.958488    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bkm\" (UniqueName: \"kubernetes.io/projected/4b032043-f55e-472f-bfae-107a66ab7aa0-kube-api-access-q2bkm\") pod \"migrator-66df5d496b-7b6st\" (UID: \"4b032043-f55e-472f-bfae-107a66ab7aa0\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-7b6st"
Oct 30 19:24:51.959973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.959955    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn899\" (UniqueName: \"kubernetes.io/projected/b0daabe8-7547-4768-9157-55e9a70f1fc1-kube-api-access-rn899\") pod \"console-operator-7cd56b8dbc-gc2t4\" (UID: \"b0daabe8-7547-4768-9157-55e9a70f1fc1\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:24:51.994241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994212    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994245    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994267    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994329    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-etc-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994329    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-log-socket\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994330    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994355    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994402    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994407    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-etc-selinux\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994427    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994456    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.994490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994477    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994499    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.994508    5085 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994519    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-multus\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994538    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-cnibin\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994580    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994541    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-systemd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994526    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-kubelet\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994617    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.994632    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.494610738 +0000 UTC m=+1.999014508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : secret "metrics-daemon-secret" not found
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994651    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-var-lib-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994704    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-sys-fs\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.994740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994721    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994750    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994764    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-dbus\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994771    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994801    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-netns\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994812    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994821    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-socket-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994839    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994856    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994871    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994892    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994898    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-systemd\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994918    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994913    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-system-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994933    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-slash\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994945    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-socket-dir-parent\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994948    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994987    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995001    5085 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995014    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995018    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/322552eb-09e0-405f-b174-a7e3317605d3-host-slash\") pod \"iptables-alerter-xs627\" (UID: \"322552eb-09e0-405f-b174-a7e3317605d3\") " pod="openshift-network-operator/iptables-alerter-xs627"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995041    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995059    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995066    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.495049039 +0000 UTC m=+1.999452827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.994988    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-host\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995074    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95cc74a0-83aa-4d4b-9fa9-b604093c4e97-host\") pod \"node-ca-4pq5w\" (UID: \"95cc74a0-83aa-4d4b-9fa9-b604093c4e97\") " pod="openshift-image-registry/node-ca-4pq5w"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995102    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995129    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995156    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995181    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995200    5085 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995204    5085 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:24:51.995221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995220    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995229    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-ovn\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995199    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/983dc1c4-4111-46f2-a526-f712eb5e56bf-hosts-file\") pod \"node-resolver-gxfnz\" (UID: \"983dc1c4-4111-46f2-a526-f712eb5e56bf\") " pod="openshift-dns/node-resolver-gxfnz"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995267    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995255    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.495238049 +0000 UTC m=+1.999641781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995300    5085 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995325    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995300    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-run-openvswitch\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995340    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.495328424 +0000 UTC m=+1.999732186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995369    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995394    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995413    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995421    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995446    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0f9c4542-7226-437e-85a9-7d7639ffbab7-crio-socket\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995450    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995473    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysconfig\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995478    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995495    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-bin\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995503    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995532    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995554    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995595    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995639    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-kubelet\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995639    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995672    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995681    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-sysctl-conf\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995690    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-var-lib-kubelet\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995646    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-lib-modules\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995702    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-node-log\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995702    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995730    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-device-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995755    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995788    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995811    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995816    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d83467d2-3710-458a-8ec7-a8cc4d2c4b3f-registration-dir\") pod \"aws-ebs-csi-driver-node-jrdw7\" (UID: \"d83467d2-3710-458a-8ec7-a8cc4d2c4b3f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jrdw7"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995839    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-modprobe-d\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995850    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d9e730e-03fe-43f4-983e-00a9e6975b6e-kubelet-config\") pod \"global-pull-secret-syncer-vg85t\" (UID: \"4d9e730e-03fe-43f4-983e-00a9e6975b6e\") " pod="kube-system/global-pull-secret-syncer-vg85t"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995856    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995885    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.995893    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-conf-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995902    5085 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995914    5085 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f75bb8685-6qjls: secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995973    5085 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995985    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls podName:fff74ec7-d916-4f36-a3e8-a77eb7649121 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.495971068 +0000 UTC m=+2.000374808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls") pod "image-registry-f75bb8685-6qjls" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121") : secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.995987    5085 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c5856f576-f75k6: secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996004    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.996039    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls podName:0c179c5b-fe07-4373-889d-2807b0e0a206 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.496025686 +0000 UTC m=+2.000429411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls") pod "image-registry-7c5856f576-f75k6" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206") : secret "image-registry-tls" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996060    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996065    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-etc-kubernetes\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996087    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996115    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.996135    5085 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996141    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996166    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.996174    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.496160253 +0000 UTC m=+2.000563979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996185    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-multus-certs\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996190    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-systemd-units\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996200    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996216    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-netns\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996227    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-multus-cni-dir\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996226    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996248    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-cnibin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-hostroot\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996268    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996295    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996321    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996344    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996347    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d0b1c6c7-c184-4847-a6b2-9dc5371171b5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6ssb\" (UID: \"d0b1c6c7-c184-4847-a6b2-9dc5371171b5\") " pod="openshift-multus/multus-additional-cni-plugins-q6ssb"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996368    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996384    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-var-lib-cni-bin\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996391    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996343    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-host-run-k8s-cni-cncf-io\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996397    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcc0e467-8e63-4365-8ef7-d6cc511ca027-etc-kubernetes\") pod \"multus-w8279\" (UID: \"dcc0e467-8e63-4365-8ef7-d6cc511ca027\") " pod="openshift-multus/multus-w8279"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996436    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996445    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-run\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996458    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996480    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996485    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-run-ovn-kubernetes\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.996452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996497    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996453    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ed874a-270d-4a83-b7d3-15d3b10454f2-sys\") pod \"tuned-g59hg\" (UID: \"02ed874a-270d-4a83-b7d3-15d3b10454f2\") " pod="openshift-cluster-node-tuning-operator/tuned-g59hg"
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996582    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e407fc34-c94f-4bfe-adf1-5ec24e48de7e-host-cni-netd\") pod \"ovnkube-node-rx5vx\" (UID: \"e407fc34-c94f-4bfe-adf1-5ec24e48de7e\") " pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:51.996598    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:52.496584885 +0000 UTC m=+2.000988641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.996992    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0f9c4542-7226-437e-85a9-7d7639ffbab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.997628    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81a6af15-f8e9-4b72-b798-ebf0438e83f4-metrics-tls\") pod \"dns-default-54xrz\" (UID: \"81a6af15-f8e9-4b72-b798-ebf0438e83f4\") " pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:51.998649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:51.997802    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b25523fa-8b51-42e3-8e6a-79af2ce1c71c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dkqt5\" (UID: \"b25523fa-8b51-42e3-8e6a-79af2ce1c71c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:52.017804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.017772    5085 scope.go:122] "RemoveContainer" containerID="1bad3c621c08749573ff4c80410dfb2daaf8a5bd7dd686cd1ae3409b13be31cb"
Oct 30 19:24:52.020637 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.020510    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"
Oct 30 19:24:52.032835 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.032806    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:52.415556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.415480    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5"]
Oct 30 19:24:52.464436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.464405    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-54xrz"]
Oct 30 19:24:52.468818 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:52.468791    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a6af15_f8e9_4b72_b798_ebf0438e83f4.slice/crio-f7b29bb9db2d16182a8b6ced1573363af5416bbbc1945fdab845f1e8bc6b53a4 WatchSource:0}: Error finding container f7b29bb9db2d16182a8b6ced1573363af5416bbbc1945fdab845f1e8bc6b53a4: Status 404 returned error can't find the container with id f7b29bb9db2d16182a8b6ced1573363af5416bbbc1945fdab845f1e8bc6b53a4
Oct 30 19:24:52.471017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.470997    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:24:52.501790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501624    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:52.501790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501685    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:52.501790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501719    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:52.501790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501765    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501795    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501870    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.501852894 +0000 UTC m=+3.006256619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501890    5085 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501915    5085 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501936    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.501922046 +0000 UTC m=+3.006325794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : secret "router-metrics-certs-default" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501939    5085 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501890    5085 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.501970    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls podName:51992eb3-1df9-4e7f-9541-a9e5c8aab9f7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.501955412 +0000 UTC m=+3.006359145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-n9x5b" (UID: "51992eb3-1df9-4e7f-9541-a9e5c8aab9f7") : secret "cluster-monitoring-operator-tls" not found
Oct 30 19:24:52.502045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.501996    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.502064    5085 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.502064    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.502104    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.502093469 +0000 UTC m=+3.006497194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.502124    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.502117929 +0000 UTC m=+3.006521654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : secret "metrics-daemon-secret" not found
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:52.502139    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:53.50213333 +0000 UTC m=+3.006537056 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:52.502325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.502172    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:52.504242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.504225    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"image-registry-7c5856f576-f75k6\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") " pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:52.504312 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.504288    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"image-registry-f75bb8685-6qjls\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") " pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:52.620866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.620836    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:52.630476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.630450    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:52.696307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.695456    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:24:52.696307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.695623    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4" event={"ID":"b0daabe8-7547-4768-9157-55e9a70f1fc1","Type":"ContainerStarted","Data":"abc7c7536a0d0f3a1bb17304cf9657cbce8e85eec05710b677d885fc4a3be5eb"}
Oct 30 19:24:52.698175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.698146    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5" event={"ID":"b25523fa-8b51-42e3-8e6a-79af2ce1c71c","Type":"ContainerStarted","Data":"5a82946bbf5ef49fb26b0a88e4951caa0e72f818c9753a65fda8592798d86aae"}
Oct 30 19:24:52.700455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:52.700429    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54xrz" event={"ID":"81a6af15-f8e9-4b72-b798-ebf0438e83f4","Type":"ContainerStarted","Data":"f7b29bb9db2d16182a8b6ced1573363af5416bbbc1945fdab845f1e8bc6b53a4"}
Oct 30 19:24:53.008409 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.008227    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:24:53.014031 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:53.014000    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff74ec7_d916_4f36_a3e8_a77eb7649121.slice/crio-eafd7f89246ffc4bc60f525b59a7dde588709c25596037b3d4000136dcb9a43d WatchSource:0}: Error finding container eafd7f89246ffc4bc60f525b59a7dde588709c25596037b3d4000136dcb9a43d: Status 404 returned error can't find the container with id eafd7f89246ffc4bc60f525b59a7dde588709c25596037b3d4000136dcb9a43d
Oct 30 19:24:53.055576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.055533    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:24:53.059855 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:53.059835    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c179c5b_fe07_4373_889d_2807b0e0a206.slice/crio-0f24e732778dd66fe5cc025df5570f978aad19deda220fb6439cdbc8e1c4fe0e WatchSource:0}: Error finding container 0f24e732778dd66fe5cc025df5570f978aad19deda220fb6439cdbc8e1c4fe0e: Status 404 returned error can't find the container with id 0f24e732778dd66fe5cc025df5570f978aad19deda220fb6439cdbc8e1c4fe0e
Oct 30 19:24:53.511397 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511361    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:53.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511410    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:53.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511473    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:53.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511523    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:53.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511554    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511600    5085 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.511634    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511672    5085 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511681    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:55.511659191 +0000 UTC m=+5.016062932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511742    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:55.511723327 +0000 UTC m=+5.016127055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511740    5085 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:24:53.511828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511792    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:55.511778393 +0000 UTC m=+5.016182132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : secret "metrics-daemon-secret" not found
Oct 30 19:24:53.512149 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:53.511850    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:55.511838222 +0000 UTC m=+5.016241950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:53.513754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.513729    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-metrics-certs\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:53.513875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.513846    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/51992eb3-1df9-4e7f-9541-a9e5c8aab9f7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-n9x5b\" (UID: \"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:53.518660 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.518644    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.707177    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" event={"ID":"0c179c5b-fe07-4373-889d-2807b0e0a206","Type":"ContainerStarted","Data":"0b92ac2d74517cbd33cd80671eb4aee6f308865243b3509367189783c0f50482"}
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.707280    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" event={"ID":"0c179c5b-fe07-4373-889d-2807b0e0a206","Type":"ContainerStarted","Data":"0f24e732778dd66fe5cc025df5570f978aad19deda220fb6439cdbc8e1c4fe0e"}
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.709411    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" event={"ID":"fff74ec7-d916-4f36-a3e8-a77eb7649121","Type":"ContainerStarted","Data":"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"}
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.709469    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" event={"ID":"fff74ec7-d916-4f36-a3e8-a77eb7649121","Type":"ContainerStarted","Data":"eafd7f89246ffc4bc60f525b59a7dde588709c25596037b3d4000136dcb9a43d"}
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.711354    5085 generic.go:356] "Generic (PLEG): container finished" podID="2f913148-3853-4762-a98a-148e2fdb44b8" containerID="f687ec8fb036d8858cae026d9ceb1f664321a091e1e18688de213c5243f96301" exitCode=0
Oct 30 19:24:53.711610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.711416    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerDied","Data":"f687ec8fb036d8858cae026d9ceb1f664321a091e1e18688de213c5243f96301"}
Oct 30 19:24:53.712291 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.711755    5085 scope.go:122] "RemoveContainer" containerID="f687ec8fb036d8858cae026d9ceb1f664321a091e1e18688de213c5243f96301"
Oct 30 19:24:53.921449 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:53.921419    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b"]
Oct 30 19:24:53.926883 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:24:53.926853    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51992eb3_1df9_4e7f_9541_a9e5c8aab9f7.slice/crio-fd5ea77c4c9badbb7443a764449c6dcddc28d62a226219e98946fe6f0b648fb7 WatchSource:0}: Error finding container fd5ea77c4c9badbb7443a764449c6dcddc28d62a226219e98946fe6f0b648fb7: Status 404 returned error can't find the container with id fd5ea77c4c9badbb7443a764449c6dcddc28d62a226219e98946fe6f0b648fb7
Oct 30 19:24:54.553502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.553463    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:24:54.718036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.717991    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerStarted","Data":"b2e2723bc7aeed5af8d39eadbb4a7f554cca2a70a2c7ee2a026058c7e0cfb8a4"}
Oct 30 19:24:54.721118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.721093    5085 generic.go:356] "Generic (PLEG): container finished" podID="ecfafb0f-0c2e-4703-be40-c9c2a080fd48" containerID="c513e96d9132cac55b6864ba2c4a16f39cfe73b6092a3f3f3faa4ead5b656c07" exitCode=0
Oct 30 19:24:54.721244 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.721158    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj" event={"ID":"ecfafb0f-0c2e-4703-be40-c9c2a080fd48","Type":"ContainerDied","Data":"c513e96d9132cac55b6864ba2c4a16f39cfe73b6092a3f3f3faa4ead5b656c07"}
Oct 30 19:24:54.721421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.721407    5085 scope.go:122] "RemoveContainer" containerID="c513e96d9132cac55b6864ba2c4a16f39cfe73b6092a3f3f3faa4ead5b656c07"
Oct 30 19:24:54.722323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.722300    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b" event={"ID":"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7","Type":"ContainerStarted","Data":"fd5ea77c4c9badbb7443a764449c6dcddc28d62a226219e98946fe6f0b648fb7"}
Oct 30 19:24:54.722430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:54.722346    5085 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.530763    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.530844    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.530880    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.530942    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531077    5085 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531137    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls podName:0f9c4542-7226-437e-85a9-7d7639ffbab7 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:59.531118702 +0000 UTC m=+9.035522442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbzm8" (UID: "0f9c4542-7226-437e-85a9-7d7639ffbab7") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531508    5085 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531551    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert podName:1e59d18a-360e-4921-a459-37f3557f9012 nodeName:}" failed. No retries permitted until 2026-10-30 19:24:59.531541231 +0000 UTC m=+9.035944968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-sltb5" (UID: "1e59d18a-360e-4921-a459-37f3557f9012") : secret "networking-console-plugin-cert" not found
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531633    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle podName:d9b9a83a-3e00-46aa-9cdd-4508a3f917eb nodeName:}" failed. No retries permitted until 2026-10-30 19:24:59.531625511 +0000 UTC m=+9.036029236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle") pod "router-default-8598d5bb4c-4xl9z" (UID: "d9b9a83a-3e00-46aa-9cdd-4508a3f917eb") : configmap references non-existent config key: service-ca.crt
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531673    5085 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:24:55.531730 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:55.531697    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs podName:1a71c312-1f67-4e38-a2dd-d54417c26e9c nodeName:}" failed. No retries permitted until 2026-10-30 19:24:59.531687085 +0000 UTC m=+9.036090828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs") pod "network-metrics-daemon-lf6jh" (UID: "1a71c312-1f67-4e38-a2dd-d54417c26e9c") : secret "metrics-daemon-secret" not found
Oct 30 19:24:55.591701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.591533    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:24:55.730741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.730704    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xv9tj" event={"ID":"ecfafb0f-0c2e-4703-be40-c9c2a080fd48","Type":"ContainerStarted","Data":"673c2d7f24e9dacc8f2b200afd8a37f82d23abc9c735b6db2c5b72107cb969fc"}
Oct 30 19:24:55.735613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.733804    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5" event={"ID":"b25523fa-8b51-42e3-8e6a-79af2ce1c71c","Type":"ContainerStarted","Data":"5ebf3cfed59100ec54c7509328b7904fa301d44d08be18e2fa261744fd28d69d"}
Oct 30 19:24:55.735613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:55.733859    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dkqt5" event={"ID":"b25523fa-8b51-42e3-8e6a-79af2ce1c71c","Type":"ContainerStarted","Data":"8628ca9b3e8189de024f938350b62f376de989a8c44e916c3f7068b4f11caf65"}
Oct 30 19:24:56.494233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:56.494204    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:56.739400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:56.739363    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54xrz" event={"ID":"81a6af15-f8e9-4b72-b798-ebf0438e83f4","Type":"ContainerStarted","Data":"e38932a56f2ee75b0897bc122bbf67dc9f1cb1e43a56d1ae2683e8d24c0e6c7b"}
Oct 30 19:24:56.739907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:56.739426    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-54xrz" event={"ID":"81a6af15-f8e9-4b72-b798-ebf0438e83f4","Type":"ContainerStarted","Data":"bf2ad7845e17860ea7afd493029586641bd6272ff2ac59b9f6dab613d1544650"}
Oct 30 19:24:56.739907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:56.739674    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-54xrz"
Oct 30 19:24:57.143402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:57.143362    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:24:57.746738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:57.746701    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-n9x5b" event={"ID":"51992eb3-1df9-4e7f-9541-a9e5c8aab9f7","Type":"ContainerStarted","Data":"10ae8afd4d46dd6e4e8ae067f30f3edfc9387f24974a0dc82396f64376a0f207"}
Oct 30 19:24:57.886636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:57.886609    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:24:57.886826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:57.886814    5085 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:24:59.567229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.567194    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:59.567229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.567238    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:59.567730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.567300    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:59.567730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.567339    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:59.567923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.567893    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9b9a83a-3e00-46aa-9cdd-4508a3f917eb-service-ca-bundle\") pod \"router-default-8598d5bb4c-4xl9z\" (UID: \"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb\") " pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:59.569656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.569635    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a71c312-1f67-4e38-a2dd-d54417c26e9c-metrics-certs\") pod \"network-metrics-daemon-lf6jh\" (UID: \"1a71c312-1f67-4e38-a2dd-d54417c26e9c\") " pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:59.569775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.569697    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1e59d18a-360e-4921-a459-37f3557f9012-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-sltb5\" (UID: \"1e59d18a-360e-4921-a459-37f3557f9012\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:59.569775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.569707    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0f9c4542-7226-437e-85a9-7d7639ffbab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbzm8\" (UID: \"0f9c4542-7226-437e-85a9-7d7639ffbab7\") " pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:24:59.756226 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.756195    5085 generic.go:356] "Generic (PLEG): container finished" podID="2f913148-3853-4762-a98a-148e2fdb44b8" containerID="b2e2723bc7aeed5af8d39eadbb4a7f554cca2a70a2c7ee2a026058c7e0cfb8a4" exitCode=0
Oct 30 19:24:59.756367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.756262    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerDied","Data":"b2e2723bc7aeed5af8d39eadbb4a7f554cca2a70a2c7ee2a026058c7e0cfb8a4"}
Oct 30 19:24:59.756367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.756310    5085 scope.go:122] "RemoveContainer" containerID="f687ec8fb036d8858cae026d9ceb1f664321a091e1e18688de213c5243f96301"
Oct 30 19:24:59.756697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.756676    5085 scope.go:122] "RemoveContainer" containerID="b2e2723bc7aeed5af8d39eadbb4a7f554cca2a70a2c7ee2a026058c7e0cfb8a4"
Oct 30 19:24:59.756908 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:24:59.756885    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-zdmh7_openshift-insights(2f913148-3853-4762-a98a-148e2fdb44b8)\"" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" podUID="2f913148-3853-4762-a98a-148e2fdb44b8"
Oct 30 19:24:59.818006 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.817938    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:24:59.821544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.821517    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5"
Oct 30 19:24:59.829595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.829575    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lf6jh"
Oct 30 19:24:59.833057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:24:59.833031    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbzm8"
Oct 30 19:25:00.175208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.175178    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rx5vx"
Oct 30 19:25:00.197274 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.197250    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-sltb5"]
Oct 30 19:25:00.229550 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:25:00.229516    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e59d18a_360e_4921_a459_37f3557f9012.slice/crio-29b0ab6b1174b9970960e10fafd924e76593c9b5e1cd8f0954651d4955a02816 WatchSource:0}: Error finding container 29b0ab6b1174b9970960e10fafd924e76593c9b5e1cd8f0954651d4955a02816: Status 404 returned error can't find the container with id 29b0ab6b1174b9970960e10fafd924e76593c9b5e1cd8f0954651d4955a02816
Oct 30 19:25:00.243900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.243881    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8598d5bb4c-4xl9z"]
Oct 30 19:25:00.244720 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:25:00.244698    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b9a83a_3e00_46aa_9cdd_4508a3f917eb.slice/crio-edef05eb40fb2a0ccda6059b2df3be56a2374285640222d1bed335cc56975d5b WatchSource:0}: Error finding container edef05eb40fb2a0ccda6059b2df3be56a2374285640222d1bed335cc56975d5b: Status 404 returned error can't find the container with id edef05eb40fb2a0ccda6059b2df3be56a2374285640222d1bed335cc56975d5b
Oct 30 19:25:00.269353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.269329    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:25:00.269455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.269444    5085 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:25:00.273132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.273111    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-gc2t4"
Oct 30 19:25:00.282171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.282147    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lf6jh"]
Oct 30 19:25:00.285869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.285851    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbzm8"]
Oct 30 19:25:00.287093 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:25:00.286979    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a71c312_1f67_4e38_a2dd_d54417c26e9c.slice/crio-a1d1c44a7b0833a2afc37b1c2a1cfe7cb868b175be3d2a9f96fe791cb3bfe43d WatchSource:0}: Error finding container a1d1c44a7b0833a2afc37b1c2a1cfe7cb868b175be3d2a9f96fe791cb3bfe43d: Status 404 returned error can't find the container with id a1d1c44a7b0833a2afc37b1c2a1cfe7cb868b175be3d2a9f96fe791cb3bfe43d
Oct 30 19:25:00.288893 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:25:00.288873    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9c4542_7226_437e_85a9_7d7639ffbab7.slice/crio-c1d1591d4e8ef23d67608ec9cf2746121baa5e455960e37c5d008980040b6536 WatchSource:0}: Error finding container c1d1591d4e8ef23d67608ec9cf2746121baa5e455960e37c5d008980040b6536: Status 404 returned error can't find the container with id c1d1591d4e8ef23d67608ec9cf2746121baa5e455960e37c5d008980040b6536
Oct 30 19:25:00.763761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.763726    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z" event={"ID":"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb","Type":"ContainerStarted","Data":"c80e5e45d87984473397e48e23f7b0b1a0d64e876a94badf1f334e8cfa0987ed"}
Oct 30 19:25:00.764377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.763783    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z" event={"ID":"d9b9a83a-3e00-46aa-9cdd-4508a3f917eb","Type":"ContainerStarted","Data":"edef05eb40fb2a0ccda6059b2df3be56a2374285640222d1bed335cc56975d5b"}
Oct 30 19:25:00.770395 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.770361    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5" event={"ID":"1e59d18a-360e-4921-a459-37f3557f9012","Type":"ContainerStarted","Data":"29b0ab6b1174b9970960e10fafd924e76593c9b5e1cd8f0954651d4955a02816"}
Oct 30 19:25:00.773026 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.773001    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lf6jh" event={"ID":"1a71c312-1f67-4e38-a2dd-d54417c26e9c","Type":"ContainerStarted","Data":"a1d1c44a7b0833a2afc37b1c2a1cfe7cb868b175be3d2a9f96fe791cb3bfe43d"}
Oct 30 19:25:00.774443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.774419    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbzm8" event={"ID":"0f9c4542-7226-437e-85a9-7d7639ffbab7","Type":"ContainerStarted","Data":"1fbd4808310ec662fb5d020e83146d412f41ed1971e47dcd726f7692a7083b64"}
Oct 30 19:25:00.774524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.774472    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbzm8" event={"ID":"0f9c4542-7226-437e-85a9-7d7639ffbab7","Type":"ContainerStarted","Data":"c1d1591d4e8ef23d67608ec9cf2746121baa5e455960e37c5d008980040b6536"}
Oct 30 19:25:00.818755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.818723    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:25:00.821576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.821539    5085 patch_prober.go:29] interesting pod/router-default-8598d5bb4c-4xl9z container/router namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[-]backend-proxy-http failed: reason withheld
Oct 30 19:25:00.821576 ip-10-0-134-200 kubenswrapper[5085]: [+]has-synced ok
Oct 30 19:25:00.821576 ip-10-0-134-200 kubenswrapper[5085]: [+]process-running ok
Oct 30 19:25:00.821576 ip-10-0-134-200 kubenswrapper[5085]: healthz check failed
Oct 30 19:25:00.821773 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:00.821609    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z" podUID="d9b9a83a-3e00-46aa-9cdd-4508a3f917eb" containerName="router" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Oct 30 19:25:01.278873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:01.278835    5085 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-200.us-west-2.compute.internal" event="NodeReady"
Oct 30 19:25:01.782301 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:01.782265    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbzm8" event={"ID":"0f9c4542-7226-437e-85a9-7d7639ffbab7","Type":"ContainerStarted","Data":"5b83d13931fd913ad2bdad1714198a8a1da30694475e81f7b856e53979f90472"}
Oct 30 19:25:01.820037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:01.820013    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:25:02.788456 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:02.788415    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-sltb5" event={"ID":"1e59d18a-360e-4921-a459-37f3557f9012","Type":"ContainerStarted","Data":"3348b5b5ef51e71e96e2ac16ac839dcfaa62d8977c5198d13511ac75e5ffefb7"}
Oct 30 19:25:02.790535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:02.790447    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lf6jh" event={"ID":"1a71c312-1f67-4e38-a2dd-d54417c26e9c","Type":"ContainerStarted","Data":"db01f401e30479bbdf0855f641deb7c122a7d9dd2e6677071e5de074b5068a5c"}
Oct 30 19:25:02.790535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:02.790501    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lf6jh" event={"ID":"1a71c312-1f67-4e38-a2dd-d54417c26e9c","Type":"ContainerStarted","Data":"7c0e3ed4fc3146b3a09be2e96d7e6fde72ebbdf109ef259904f4d60b31d5542b"}
Oct 30 19:25:03.570333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:03.570302    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:25:03.571459 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:03.571438    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8598d5bb4c-4xl9z"
Oct 30 19:25:03.797820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:03.797786    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbzm8" event={"ID":"0f9c4542-7226-437e-85a9-7d7639ffbab7","Type":"ContainerStarted","Data":"2c9113f53c119ea7a0f35c3ac92f443f834a3c56cadf51b010cb93ae09420116"}
Oct 30 19:25:05.948847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:05.948811    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhxdt"
Oct 30 19:25:06.748864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:06.748832    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-54xrz"
Oct 30 19:25:09.429600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:09.429550    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:25:09.435255 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:09.435134    5085 patch_prober.go:29] interesting pod/image-registry-f75bb8685-6qjls container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 19:25:09.435255 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:09.435194    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:25:10.691317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:10.691287    5085 patch_prober.go:29] interesting pod/image-registry-7c5856f576-f75k6 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 19:25:10.691781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:10.691342    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:25:12.623721 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:12.623689    5085 scope.go:122] "RemoveContainer" containerID="b2e2723bc7aeed5af8d39eadbb4a7f554cca2a70a2c7ee2a026058c7e0cfb8a4"
Oct 30 19:25:12.849635 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:12.849597    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zdmh7" event={"ID":"2f913148-3853-4762-a98a-148e2fdb44b8","Type":"ContainerStarted","Data":"1fbeb126039b2784fe84b33b0c30c2613acff5414f0bb183c5b4390c9404ebaa"}
Oct 30 19:25:15.594963 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:15.594933    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cjw9x"
Oct 30 19:25:15.739086 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:15.739059    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:25:19.435649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:19.435621    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:25:21.172556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.172520    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rkrcl"]
Oct 30 19:25:21.200082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.200046    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.250808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.250774    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:25:21.251089 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.251073    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8mnv5\""
Oct 30 19:25:21.296928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.296901    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:25:21.343680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.343645    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:25:21.343832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.343777    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:25:21.355134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355106    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-root\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355157    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gspw\" (UniqueName: \"kubernetes.io/projected/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-kube-api-access-9gspw\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355228    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-tls\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355252    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-metrics-client-ca\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355275    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355339    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-sys\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355392    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-textfile\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355462    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-wtmp\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.355509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.355480    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456028 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.455940    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-root\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456028 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456001    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gspw\" (UniqueName: \"kubernetes.io/projected/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-kube-api-access-9gspw\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456029    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-tls\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456052    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-metrics-client-ca\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456081    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456048    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-root\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456118    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-sys\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456142    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-textfile\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456186    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-wtmp\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456209    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456309    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-wtmp\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456173    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-sys\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456646    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-textfile\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456799    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-metrics-client-ca\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.456857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.456816    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-accelerators-collector-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.458452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.458432    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-tls\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.458548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.458531    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.516219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.516180    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gspw\" (UniqueName: \"kubernetes.io/projected/44fc5a64-c15d-4da6-b787-7a3fbe86f12e-kube-api-access-9gspw\") pod \"node-exporter-rkrcl\" (UID: \"44fc5a64-c15d-4da6-b787-7a3fbe86f12e\") " pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.549265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.549233    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rkrcl"
Oct 30 19:25:21.588199 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:25:21.588169    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fc5a64_c15d_4da6_b787_7a3fbe86f12e.slice/crio-26871217a6d0d3f102855f079adb6bd3282bb926ddd91eda3cec1a16f394ba1c WatchSource:0}: Error finding container 26871217a6d0d3f102855f079adb6bd3282bb926ddd91eda3cec1a16f394ba1c: Status 404 returned error can't find the container with id 26871217a6d0d3f102855f079adb6bd3282bb926ddd91eda3cec1a16f394ba1c
Oct 30 19:25:21.891667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:21.891589    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkrcl" event={"ID":"44fc5a64-c15d-4da6-b787-7a3fbe86f12e","Type":"ContainerStarted","Data":"26871217a6d0d3f102855f079adb6bd3282bb926ddd91eda3cec1a16f394ba1c"}
Oct 30 19:25:21.937973 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:25:21.937935    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:25:23.901676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:23.901645    5085 generic.go:356] "Generic (PLEG): container finished" podID="44fc5a64-c15d-4da6-b787-7a3fbe86f12e" containerID="9707a7ca3713520a84e3b23f9080ac377085c1508181857e097d5dcdfb5a48bb" exitCode=0
Oct 30 19:25:23.902009 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:23.901724    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkrcl" event={"ID":"44fc5a64-c15d-4da6-b787-7a3fbe86f12e","Type":"ContainerDied","Data":"9707a7ca3713520a84e3b23f9080ac377085c1508181857e097d5dcdfb5a48bb"}
Oct 30 19:25:24.907324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:24.907288    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkrcl" event={"ID":"44fc5a64-c15d-4da6-b787-7a3fbe86f12e","Type":"ContainerStarted","Data":"3059b17e41f4b4341f15e952386e1538fe85d8e287e71bce05be15a8f6dbfbf6"}
Oct 30 19:25:24.907718 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:24.907339    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rkrcl" event={"ID":"44fc5a64-c15d-4da6-b787-7a3fbe86f12e","Type":"ContainerStarted","Data":"7260a6365ea79512edd77298ea01a10d08b45f6535b881a7a4be4b8feba335fa"}
Oct 30 19:25:25.021426 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:25.021354    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rkrcl" podStartSLOduration=1.8583364310000001 podStartE2EDuration="4.021336878s" totalImagesPullingTime="2.089609611s" totalInitContainerRuntime="73.390836ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:25:25.020663267 +0000 UTC m=+34.525067036" watchObservedRunningTime="2026-10-30 19:25:25.021336878 +0000 UTC m=+34.525740626"
Oct 30 19:25:31.842245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:31.842165    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:25:34.456993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.456947    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerName="registry" containerID="cri-o://e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded" gracePeriod=30
Oct 30 19:25:34.712357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.712308    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:25:34.872803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872776    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872827    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872859    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872876    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872900    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872930    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872956    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.872993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.872992    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") pod \"fff74ec7-d916-4f36-a3e8-a77eb7649121\" (UID: \"fff74ec7-d916-4f36-a3e8-a77eb7649121\") "
Oct 30 19:25:34.873778 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.873693    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:34.873778 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.873760    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:34.883420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.883380    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:34.883990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.883774    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:34.883990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.883778    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:34.883990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.883674    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "kube-api-access-m5qr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:34.883990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.883656    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:34.886497 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.886472    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted" pod "fff74ec7-d916-4f36-a3e8-a77eb7649121" (UID: "fff74ec7-d916-4f36-a3e8-a77eb7649121"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:34.945280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.945246    5085 generic.go:356] "Generic (PLEG): container finished" podID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerID="e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded" exitCode=0
Oct 30 19:25:34.945444 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.945301    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f75bb8685-6qjls"
Oct 30 19:25:34.945444 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.945308    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" event={"ID":"fff74ec7-d916-4f36-a3e8-a77eb7649121","Type":"ContainerDied","Data":"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"}
Oct 30 19:25:34.945444 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.945368    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f75bb8685-6qjls" event={"ID":"fff74ec7-d916-4f36-a3e8-a77eb7649121","Type":"ContainerDied","Data":"eafd7f89246ffc4bc60f525b59a7dde588709c25596037b3d4000136dcb9a43d"}
Oct 30 19:25:34.945444 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.945388    5085 scope.go:122] "RemoveContainer" containerID="e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"
Oct 30 19:25:34.954376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.954359    5085 scope.go:122] "RemoveContainer" containerID="e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"
Oct 30 19:25:34.954655 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:25:34.954634    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded\": container with ID starting with e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded not found: ID does not exist" containerID="e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"
Oct 30 19:25:34.954720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.954663    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded"} err="failed to get container status \"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded\": rpc error: code = NotFound desc = could not find container \"e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded\": container with ID starting with e05fa9202897fa8cb07ac1737cd92cf37e2d0cce3b52919401f4eb22f248eded not found: ID does not exist"
Oct 30 19:25:34.974113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974057    5085 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-bound-sa-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974080    5085 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974091    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qr2\" (UniqueName: \"kubernetes.io/projected/fff74ec7-d916-4f36-a3e8-a77eb7649121-kube-api-access-m5qr2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974100    5085 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-trusted-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974110    5085 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fff74ec7-d916-4f36-a3e8-a77eb7649121-ca-trust-extracted\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974119    5085 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fff74ec7-d916-4f36-a3e8-a77eb7649121-registry-certificates\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974129    5085 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-installation-pull-secrets\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:34.974335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:34.974138    5085 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fff74ec7-d916-4f36-a3e8-a77eb7649121-image-registry-private-configuration\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:35.108361 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:35.108326    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:25:35.117834 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:35.117809    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f75bb8685-6qjls"]
Oct 30 19:25:36.629444 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:36.629412    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" path="/var/lib/kubelet/pods/fff74ec7-d916-4f36-a3e8-a77eb7649121/volumes"
Oct 30 19:25:41.841883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:41.841851    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:25:41.842289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:41.842065    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2md8m-pt29t" podUID="6871932b-3edd-47fd-9dcb-bc67f686b852" containerName="insights-gathering" containerID="cri-o://65c6d4312d3a632d1f80acaa2c3cb877db6965d15d8ea1e388859e3f28dd422c" gracePeriod=30
Oct 30 19:25:41.976436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:41.976406    5085 generic.go:356] "Generic (PLEG): container finished" podID="6871932b-3edd-47fd-9dcb-bc67f686b852" containerID="65c6d4312d3a632d1f80acaa2c3cb877db6965d15d8ea1e388859e3f28dd422c" exitCode=2
Oct 30 19:25:41.976590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:41.976463    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2md8m-pt29t" event={"ID":"6871932b-3edd-47fd-9dcb-bc67f686b852","Type":"ContainerDied","Data":"65c6d4312d3a632d1f80acaa2c3cb877db6965d15d8ea1e388859e3f28dd422c"}
Oct 30 19:25:42.107453 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.107422    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:25:42.123280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123253    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") pod \"6871932b-3edd-47fd-9dcb-bc67f686b852\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") "
Oct 30 19:25:42.123388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123311    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") pod \"6871932b-3edd-47fd-9dcb-bc67f686b852\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") "
Oct 30 19:25:42.123388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123360    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") pod \"6871932b-3edd-47fd-9dcb-bc67f686b852\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") "
Oct 30 19:25:42.123388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123379    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") pod \"6871932b-3edd-47fd-9dcb-bc67f686b852\" (UID: \"6871932b-3edd-47fd-9dcb-bc67f686b852\") "
Oct 30 19:25:42.123774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123749    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path" pod "6871932b-3edd-47fd-9dcb-bc67f686b852" (UID: "6871932b-3edd-47fd-9dcb-bc67f686b852"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:42.123860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123747    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle" pod "6871932b-3edd-47fd-9dcb-bc67f686b852" (UID: "6871932b-3edd-47fd-9dcb-bc67f686b852"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:42.123860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.123816    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle" pod "6871932b-3edd-47fd-9dcb-bc67f686b852" (UID: "6871932b-3edd-47fd-9dcb-bc67f686b852"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:42.125910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.125875    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87" pod "6871932b-3edd-47fd-9dcb-bc67f686b852" (UID: "6871932b-3edd-47fd-9dcb-bc67f686b852"). InnerVolumeSpecName "kube-api-access-jvw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:42.223868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.223834    5085 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-service-ca-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:42.223868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.223865    5085 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6871932b-3edd-47fd-9dcb-bc67f686b852-archives-path\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:42.224072 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.223880    5085 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6871932b-3edd-47fd-9dcb-bc67f686b852-trusted-ca-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:42.224072 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.223893    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvw87\" (UniqueName: \"kubernetes.io/projected/6871932b-3edd-47fd-9dcb-bc67f686b852-kube-api-access-jvw87\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:42.981877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.981850    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2md8m-pt29t"
Oct 30 19:25:42.982268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.981849    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2md8m-pt29t" event={"ID":"6871932b-3edd-47fd-9dcb-bc67f686b852","Type":"ContainerDied","Data":"b2ddb2c47a8c6d7869bd5db6cd41b16b23a14e1f7de233490b5a581aed4b1b04"}
Oct 30 19:25:42.982268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:42.981979    5085 scope.go:122] "RemoveContainer" containerID="65c6d4312d3a632d1f80acaa2c3cb877db6965d15d8ea1e388859e3f28dd422c"
Oct 30 19:25:43.152137 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:43.152099    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:25:43.198723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:43.198687    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2md8m-pt29t"]
Oct 30 19:25:44.629149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:44.629115    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6871932b-3edd-47fd-9dcb-bc67f686b852" path="/var/lib/kubelet/pods/6871932b-3edd-47fd-9dcb-bc67f686b852/volumes"
Oct 30 19:25:51.937697 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:25:51.937654    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:25:56.862084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:56.862040    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerName="registry" containerID="cri-o://0b92ac2d74517cbd33cd80671eb4aee6f308865243b3509367189783c0f50482" gracePeriod=30
Oct 30 19:25:57.040673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.040642    5085 generic.go:356] "Generic (PLEG): container finished" podID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerID="0b92ac2d74517cbd33cd80671eb4aee6f308865243b3509367189783c0f50482" exitCode=0
Oct 30 19:25:57.040813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.040717    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" event={"ID":"0c179c5b-fe07-4373-889d-2807b0e0a206","Type":"ContainerDied","Data":"0b92ac2d74517cbd33cd80671eb4aee6f308865243b3509367189783c0f50482"}
Oct 30 19:25:57.123864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.123780    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:25:57.233012 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.232975    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233022    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233078    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233104    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233147    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233176    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233196    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233225    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") pod \"0c179c5b-fe07-4373-889d-2807b0e0a206\" (UID: \"0c179c5b-fe07-4373-889d-2807b0e0a206\") "
Oct 30 19:25:57.233600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.233552    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:57.234117 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.234076    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:57.235822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.235779    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:57.235940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.235927    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:57.236167 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.236137    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "kube-api-access-7w26d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:57.236238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.236208    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:57.236238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.236217    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:57.241032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.240916    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted" pod "0c179c5b-fe07-4373-889d-2807b0e0a206" (UID: "0c179c5b-fe07-4373-889d-2807b0e0a206"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:57.334801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334762    5085 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-trusted-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.334801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334793    5085 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-bound-sa-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.334801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334807    5085 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-installation-pull-secrets\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.335047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334824    5085 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.335047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334838    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w26d\" (UniqueName: \"kubernetes.io/projected/0c179c5b-fe07-4373-889d-2807b0e0a206-kube-api-access-7w26d\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.335047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334850    5085 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c179c5b-fe07-4373-889d-2807b0e0a206-ca-trust-extracted\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.335047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334862    5085 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c179c5b-fe07-4373-889d-2807b0e0a206-registry-certificates\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:57.335047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:57.334875    5085 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c179c5b-fe07-4373-889d-2807b0e0a206-image-registry-private-configuration\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:58.046464 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.046433    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c5856f576-f75k6"
Oct 30 19:25:58.046464 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.046456    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c5856f576-f75k6" event={"ID":"0c179c5b-fe07-4373-889d-2807b0e0a206","Type":"ContainerDied","Data":"0f24e732778dd66fe5cc025df5570f978aad19deda220fb6439cdbc8e1c4fe0e"}
Oct 30 19:25:58.046993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.046505    5085 scope.go:122] "RemoveContainer" containerID="0b92ac2d74517cbd33cd80671eb4aee6f308865243b3509367189783c0f50482"
Oct 30 19:25:58.248806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.248771    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:25:58.295416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.295380    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c5856f576-f75k6"]
Oct 30 19:25:58.629059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:25:58.629031    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" path="/var/lib/kubelet/pods/0c179c5b-fe07-4373-889d-2807b0e0a206/volumes"
Oct 30 19:26:20.136900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:26:20.136859    5085 generic.go:356] "Generic (PLEG): container finished" podID="68246332-14ef-4b27-b072-c6141c820f53" containerID="f76b0934ab21df34ec8959763c73335c6435d72d938c23ffbc347f1005b6eee1" exitCode=0
Oct 30 19:26:20.137307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:26:20.136931    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8" event={"ID":"68246332-14ef-4b27-b072-c6141c820f53","Type":"ContainerDied","Data":"f76b0934ab21df34ec8959763c73335c6435d72d938c23ffbc347f1005b6eee1"}
Oct 30 19:26:20.137307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:26:20.137271    5085 scope.go:122] "RemoveContainer" containerID="f76b0934ab21df34ec8959763c73335c6435d72d938c23ffbc347f1005b6eee1"
Oct 30 19:26:21.143087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:26:21.143053    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zrzg8" event={"ID":"68246332-14ef-4b27-b072-c6141c820f53","Type":"ContainerStarted","Data":"6bbe97b18191d942d293c35aa071a42e07d1c8f515fb69a982f63b8ecef08f5d"}
Oct 30 19:26:21.937755 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:26:21.937723    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:26:51.939069 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:26:51.939029    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:27:21.937352 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:27:21.937314    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:27:51.937484 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:27:51.937444    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:28:21.937981 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:28:21.937939    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:28:51.938181 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:28:51.938103    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:29:21.938387 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:29:21.938348    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:29:50.642706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:29:50.642673    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:29:50.644113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:29:50.644089    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:29:51.937408 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:29:51.937365    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:30:21.937908 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:30:21.937868    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:30:51.937292 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:30:51.937253    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:31:21.938749 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:31:21.938677    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:31:22.931712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.931681    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"]
Oct 30 19:31:22.931976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.931964    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerName="registry"
Oct 30 19:31:22.932019 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.931978    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerName="registry"
Oct 30 19:31:22.932019 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.931992    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6871932b-3edd-47fd-9dcb-bc67f686b852" containerName="insights-gathering"
Oct 30 19:31:22.932019 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.931998    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6871932b-3edd-47fd-9dcb-bc67f686b852" containerName="insights-gathering"
Oct 30 19:31:22.932019 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.932018    5085 cpu_manager.go:414] "containerMap: removing container" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerName="registry"
Oct 30 19:31:22.932139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.932024    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerName="registry"
Oct 30 19:31:22.932139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.932071    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c179c5b-fe07-4373-889d-2807b0e0a206" containerName="registry"
Oct 30 19:31:22.932139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.932080    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6871932b-3edd-47fd-9dcb-bc67f686b852" containerName="insights-gathering"
Oct 30 19:31:22.932139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.932089    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff74ec7-d916-4f36-a3e8-a77eb7649121" containerName="registry"
Oct 30 19:31:22.935138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.935120    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:22.973701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.973669    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:22.974063 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.973781    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9zf\" (UniqueName: \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:22.974063 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:22.973826    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.031404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.031363    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 19:31:23.031404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.031393    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bp2mm\""
Oct 30 19:31:23.031638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.031438    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 19:31:23.032622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.032602    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"]
Oct 30 19:31:23.075048 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.075017    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.075048 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.075049    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.075232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.075133    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9zf\" (UniqueName: \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.075499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.075480    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.075499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.075491    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.134515 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.134482    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9zf\" (UniqueName: \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.261125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.261050    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:31:23.625896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.625826    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"]
Oct 30 19:31:23.629235 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:23.629202    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd985353a_0b0e_4e03_87dc_7478de4d2ade.slice/crio-758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2 WatchSource:0}: Error finding container 758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2: Status 404 returned error can't find the container with id 758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2
Oct 30 19:31:23.631191 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:23.631175    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:31:24.200696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:24.200660    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerStarted","Data":"758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2"}
Oct 30 19:31:25.145612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.145552    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"]
Oct 30 19:31:25.149188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.149160    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.190072 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.190039    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.190214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.190112    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.190276 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.190239    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjwhh\" (UniqueName: \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.245203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.245170    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lkllk\""
Oct 30 19:31:25.246543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.246511    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"]
Oct 30 19:31:25.291524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.291491    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.291757 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.291548    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.291757 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.291603    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjwhh\" (UniqueName: \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.291983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.291960    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.292161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.292144    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.323007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.322973    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"]
Oct 30 19:31:25.326462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.326441    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.376784 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.376756    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-smnm9\""
Oct 30 19:31:25.379386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.379351    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjwhh\" (UniqueName: \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.387900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.387870    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"]
Oct 30 19:31:25.392704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.392007    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.392704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.392059    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.392704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.392085    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzvnk\" (UniqueName: \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.453402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.453374    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"]
Oct 30 19:31:25.456937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.456915    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.467663 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.467640    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:31:25.492886 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.492856    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.493034 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.492924    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.493114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493088    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.493184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493136    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.493184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493169    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzvnk\" (UniqueName: \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.493298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493199    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.493540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493517    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.493654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.493584    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.518804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.518773    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"]
Oct 30 19:31:25.518970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.518906    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-pv4zt\""
Oct 30 19:31:25.545618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.545588    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"]
Oct 30 19:31:25.550108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.550081    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.554153 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.554107    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzvnk\" (UniqueName: \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.593746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593712    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.593885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593771    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.593885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593842    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7vpk\" (UniqueName: \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.593996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593916    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.593996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593940    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.593996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.593984    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.594335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.594310    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.594465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.594380    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.619846 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.619819    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5xmkw\""
Oct 30 19:31:25.639083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.639060    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"]
Oct 30 19:31:25.645446 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.645422    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:31:25.653180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.653157    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.695040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.695006    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.695212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.695080    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.695212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.695102    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7vpk\" (UniqueName: \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.695426 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.695405    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.695678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.695658    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.755621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.755590    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7vpk\" (UniqueName: \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:25.776349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.776319    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:31:25.858260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.858218    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"]
Oct 30 19:31:25.861914 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:25.861886    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebd25e5_f389_43bb_b01f_f98896e872c5.slice/crio-dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0 WatchSource:0}: Error finding container dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0: Status 404 returned error can't find the container with id dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0
Oct 30 19:31:25.878095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:25.878066    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:31:26.027916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.027843    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"]
Oct 30 19:31:26.031465 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:26.031427    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c9d0d6_0003_445d_b928_e38da16560a8.slice/crio-aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8 WatchSource:0}: Error finding container aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8: Status 404 returned error can't find the container with id aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8
Oct 30 19:31:26.159984 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.159950    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"]
Oct 30 19:31:26.163585 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:26.163525    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbdda86_eb09_4f7c_95ab_5e1980209f1b.slice/crio-777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d WatchSource:0}: Error finding container 777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d: Status 404 returned error can't find the container with id 777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d
Oct 30 19:31:26.212452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.212416    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh" event={"ID":"0dbdda86-eb09-4f7c-95ab-5e1980209f1b","Type":"ContainerStarted","Data":"777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d"}
Oct 30 19:31:26.213877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.213849    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5" event={"ID":"4ebd25e5-f389-43bb-b01f-f98896e872c5","Type":"ContainerStarted","Data":"dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0"}
Oct 30 19:31:26.215332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.215298    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerStarted","Data":"aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8"}
Oct 30 19:31:26.249941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.249909    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"]
Oct 30 19:31:26.253259 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:26.253226    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022fdc57_450c_4bdd_ac98_27ab831eafed.slice/crio-ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409 WatchSource:0}: Error finding container ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409: Status 404 returned error can't find the container with id ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409
Oct 30 19:31:26.750391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.750351    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"]
Oct 30 19:31:26.755534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.755508    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.804500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.804463    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ck7\" (UniqueName: \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.804688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.804601    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.804688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.804676    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.807572 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.807540    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-8j8v5\""
Oct 30 19:31:26.810602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.810578    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"]
Oct 30 19:31:26.905593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.905547    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.905743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.905626    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ck7\" (UniqueName: \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.905743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.905675    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.906011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.905983    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.906011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.906007    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:26.966416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:26.966378    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ck7\" (UniqueName: \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:27.073773 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.073688    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:31:27.204930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.204894    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"]
Oct 30 19:31:27.209269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.209241    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.228232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.228127    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq" event={"ID":"022fdc57-450c-4bdd-ac98-27ab831eafed","Type":"ContainerStarted","Data":"ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409"}
Oct 30 19:31:27.264188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.264153    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-8jhg8\""
Oct 30 19:31:27.265738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.265708    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"]
Oct 30 19:31:27.310430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.310393    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.310634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.310448    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5c72\" (UniqueName: \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.310634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.310477    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.411585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.411490    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.411760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.411598    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.411760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.411657    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5c72\" (UniqueName: \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.412325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.412301    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.412581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.412547    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.478943 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.478821    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"]
Oct 30 19:31:27.482203 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:27.482171    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f96535_08a5_4d56_a73c_59c8d5a18ebf.slice/crio-8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992 WatchSource:0}: Error finding container 8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992: Status 404 returned error can't find the container with id 8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992
Oct 30 19:31:27.489071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.489044    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5c72\" (UniqueName: \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.533545 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.533506    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:27.961760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:27.961370    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"]
Oct 30 19:31:27.964746 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:27.964710    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1d6e7b_b2c4_441a_9261_0d0977544f6f.slice/crio-3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a WatchSource:0}: Error finding container 3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a: Status 404 returned error can't find the container with id 3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a
Oct 30 19:31:28.114056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.112315    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"]
Oct 30 19:31:28.118158 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.117518    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.167899 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.167712    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:31:28.169078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.168827    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-qqm85\""
Oct 30 19:31:28.176683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.176602    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"]
Oct 30 19:31:28.218947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.218861    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd27053-72dc-4215-885b-32f291f98853-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.218947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.218903    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.218947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.218928    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.218981    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.219017    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbmnx\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-kube-api-access-pbmnx\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.219038    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.219057    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.219075    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.219214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.219122    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd27053-72dc-4215-885b-32f291f98853-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.237834 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.237803    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh" event={"ID":"a3f96535-08a5-4d56-a73c-59c8d5a18ebf","Type":"ContainerStarted","Data":"8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992"}
Oct 30 19:31:28.241476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.241442    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerStarted","Data":"3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a"}
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320456    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd27053-72dc-4215-885b-32f291f98853-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320517    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd27053-72dc-4215-885b-32f291f98853-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320554    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320596    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320647    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320684    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbmnx\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-kube-api-access-pbmnx\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320718    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320748    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.320776    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.321248    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.323013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.321935    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd27053-72dc-4215-885b-32f291f98853-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.325386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.323384    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.325386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.323667    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.325386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.324163    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.327384 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.327356    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd27053-72dc-4215-885b-32f291f98853-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.327672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.327651    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd27053-72dc-4215-885b-32f291f98853-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.382943 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.382902    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.387233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.387203    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbmnx\" (UniqueName: \"kubernetes.io/projected/dbd27053-72dc-4215-885b-32f291f98853-kube-api-access-pbmnx\") pod \"openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg\" (UID: \"dbd27053-72dc-4215-885b-32f291f98853\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.437227 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.436910    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:28.478118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.476397    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"]
Oct 30 19:31:28.481086 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.480992    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522522 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522485    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522548    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522619    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eff08b-642e-4125-90e1-88ea8adf5385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522663    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5v5\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-kube-api-access-5w5v5\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522701    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522759    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522788    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522815    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.522968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.522851    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.533837 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.533789    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-q8znj\""
Oct 30 19:31:28.548283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.548248    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"]
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627048    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627099    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eff08b-642e-4125-90e1-88ea8adf5385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627128    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5v5\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-kube-api-access-5w5v5\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627156    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627194    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627219    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627246    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627275    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627331    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.627737    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.628328    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.629486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.628918    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7eff08b-642e-4125-90e1-88ea8adf5385-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.630774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.630478    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.630774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.630526    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.631487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.631442    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.633493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.633456    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.690983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.690917    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.691302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.691262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5v5\" (UniqueName: \"kubernetes.io/projected/f7eff08b-642e-4125-90e1-88ea8adf5385-kube-api-access-5w5v5\") pod \"maas-default-gateway-openshift-default-694db55c87-mcbsf\" (UID: \"f7eff08b-642e-4125-90e1-88ea8adf5385\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.833722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.833117    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:28.884343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:28.884253    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"]
Oct 30 19:31:28.901351 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:31:28.901149    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd27053_72dc_4215_885b_32f291f98853.slice/crio-51b89b71fb87a150bd000ac62bbf78014c8ae810f628c8ce1769e5a72e868397 WatchSource:0}: Error finding container 51b89b71fb87a150bd000ac62bbf78014c8ae810f628c8ce1769e5a72e868397: Status 404 returned error can't find the container with id 51b89b71fb87a150bd000ac62bbf78014c8ae810f628c8ce1769e5a72e868397
Oct 30 19:31:29.233873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:29.233818    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"]
Oct 30 19:31:29.259242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:29.257343    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg" event={"ID":"dbd27053-72dc-4215-885b-32f291f98853","Type":"ContainerStarted","Data":"51b89b71fb87a150bd000ac62bbf78014c8ae810f628c8ce1769e5a72e868397"}
Oct 30 19:31:30.270258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:30.270218    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" event={"ID":"f7eff08b-642e-4125-90e1-88ea8adf5385","Type":"ContainerStarted","Data":"1ebd6cd76e3bc44d6f137ba54e4559689bd9b09b66616acaa7fcbaab7024a343"}
Oct 30 19:31:38.353546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:38.353501    5085 generic.go:356] "Generic (PLEG): container finished" podID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerID="c4a89f8e2f0e64aadf17ba9cb087dbf07a6fa7f6eeaf07e32db93a8ae27769c1" exitCode=0
Oct 30 19:31:38.354038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:38.353679    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerDied","Data":"c4a89f8e2f0e64aadf17ba9cb087dbf07a6fa7f6eeaf07e32db93a8ae27769c1"}
Oct 30 19:31:38.555653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:38.554718    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:38.555653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:38.554792    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:38.555653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:38.554824    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:39.126273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.125579    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:39.126273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.125673    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:39.126273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.125717    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:31:39.366718 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.366683    5085 generic.go:356] "Generic (PLEG): container finished" podID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerID="cd2615e25f583d0dc59e061bdebf569210020ba766b37f13bf7de50190e4fc4d" exitCode=0
Oct 30 19:31:39.367167 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.366781    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh" event={"ID":"a3f96535-08a5-4d56-a73c-59c8d5a18ebf","Type":"ContainerDied","Data":"cd2615e25f583d0dc59e061bdebf569210020ba766b37f13bf7de50190e4fc4d"}
Oct 30 19:31:39.373371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.372742    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" event={"ID":"f7eff08b-642e-4125-90e1-88ea8adf5385","Type":"ContainerStarted","Data":"2de2d7f4258015d7465c595f86f0fd8f647852485ca89bf7fb1a43d652f49887"}
Oct 30 19:31:39.375808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.375785    5085 generic.go:356] "Generic (PLEG): container finished" podID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerID="128ebc96bba97935d65bd458d196298bd1c2b71ee41e9074b9830f31a4d026f1" exitCode=0
Oct 30 19:31:39.375907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.375848    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerDied","Data":"128ebc96bba97935d65bd458d196298bd1c2b71ee41e9074b9830f31a4d026f1"}
Oct 30 19:31:39.379751 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.379727    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg" event={"ID":"dbd27053-72dc-4215-885b-32f291f98853","Type":"ContainerStarted","Data":"8a3e7f6c6da1db0fb53d1d65abb79cad92ccfeb1910dabf4a91a2e081e11bb2d"}
Oct 30 19:31:39.438351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.438266    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:39.612258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.612203    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg" podStartSLOduration=1.3955030370000001 podStartE2EDuration="11.612183104s" totalImagesPullingTime="10.216680067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.610598207 +0000 UTC m=+409.115001971" watchObservedRunningTime="2026-10-30 19:31:39.612183104 +0000 UTC m=+409.116586853"
Oct 30 19:31:39.834544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.834467    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:39.837976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.837797    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podStartSLOduration=2.540613974 podStartE2EDuration="11.837779804s" totalImagesPullingTime="9.29716583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.837178427 +0000 UTC m=+409.341582177" watchObservedRunningTime="2026-10-30 19:31:39.837779804 +0000 UTC m=+409.342183551"
Oct 30 19:31:39.840599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:39.840386    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:40.385621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:40.385545    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:40.386413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:40.386392    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:31:40.443536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:40.443505    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:41.392816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.392435    5085 generic.go:356] "Generic (PLEG): container finished" podID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerID="835260ca68566ff26f3de4b6f0d9567562b9f2a0df5c915891e74f2b912062c8" exitCode=0
Oct 30 19:31:41.392816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.392484    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh" event={"ID":"0dbdda86-eb09-4f7c-95ab-5e1980209f1b","Type":"ContainerDied","Data":"835260ca68566ff26f3de4b6f0d9567562b9f2a0df5c915891e74f2b912062c8"}
Oct 30 19:31:41.396807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.396125    5085 generic.go:356] "Generic (PLEG): container finished" podID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerID="56658ee5e72743c2ab0c41a8a1d6906fe45b5af79ecfe778bfbe0e89107836a5" exitCode=0
Oct 30 19:31:41.396807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.396210    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5" event={"ID":"4ebd25e5-f389-43bb-b01f-f98896e872c5","Type":"ContainerDied","Data":"56658ee5e72743c2ab0c41a8a1d6906fe45b5af79ecfe778bfbe0e89107836a5"}
Oct 30 19:31:41.398920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.398752    5085 generic.go:356] "Generic (PLEG): container finished" podID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerID="764c32b8468b0c5fc83fae9b5be36a6d5d819222c943999c6d6875dd1b555a32" exitCode=0
Oct 30 19:31:41.398920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.398861    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerDied","Data":"764c32b8468b0c5fc83fae9b5be36a6d5d819222c943999c6d6875dd1b555a32"}
Oct 30 19:31:41.400782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.400756    5085 generic.go:356] "Generic (PLEG): container finished" podID="17c9d0d6-0003-445d-b928-e38da16560a8" containerID="35ce4882f72cff0eaf74404c25ad23cb4416c8b4bc6c8fd909c5907d2f9b008e" exitCode=0
Oct 30 19:31:41.400901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.400877    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerDied","Data":"35ce4882f72cff0eaf74404c25ad23cb4416c8b4bc6c8fd909c5907d2f9b008e"}
Oct 30 19:31:41.403868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.403845    5085 generic.go:356] "Generic (PLEG): container finished" podID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerID="a554fc56ee72edeaf23270cda45d57b9736e0d12c1db0f9cdd25c19147b653b2" exitCode=0
Oct 30 19:31:41.405818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.404932    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq" event={"ID":"022fdc57-450c-4bdd-ac98-27ab831eafed","Type":"ContainerDied","Data":"a554fc56ee72edeaf23270cda45d57b9736e0d12c1db0f9cdd25c19147b653b2"}
Oct 30 19:31:41.405818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.405686    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:41.406724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:41.406703    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg"
Oct 30 19:31:43.423087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.423044    5085 generic.go:356] "Generic (PLEG): container finished" podID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerID="1a94f16c86e533e3c7540b45baf92881ae75c2a484cc49df76bc14b86c81b302" exitCode=0
Oct 30 19:31:43.423609 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.423117    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerDied","Data":"1a94f16c86e533e3c7540b45baf92881ae75c2a484cc49df76bc14b86c81b302"}
Oct 30 19:31:43.425506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.425478    5085 generic.go:356] "Generic (PLEG): container finished" podID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerID="448bebd8e6f0e9b619c3cfb22a3a3ce141815a45a2add73f452c59173f5433b8" exitCode=0
Oct 30 19:31:43.425755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.425733    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq" event={"ID":"022fdc57-450c-4bdd-ac98-27ab831eafed","Type":"ContainerDied","Data":"448bebd8e6f0e9b619c3cfb22a3a3ce141815a45a2add73f452c59173f5433b8"}
Oct 30 19:31:43.428366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.428344    5085 generic.go:356] "Generic (PLEG): container finished" podID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerID="f809531a4aa5995011c692d0795856d3cb37e868cb66fd8967025ce89d044161" exitCode=0
Oct 30 19:31:43.428507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.428478    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh" event={"ID":"a3f96535-08a5-4d56-a73c-59c8d5a18ebf","Type":"ContainerDied","Data":"f809531a4aa5995011c692d0795856d3cb37e868cb66fd8967025ce89d044161"}
Oct 30 19:31:43.430739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.430720    5085 generic.go:356] "Generic (PLEG): container finished" podID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerID="d36b34ba2478145ebd26c42e41db27bb3e6a154ee3a1d0b9e4979bb5ad4081af" exitCode=0
Oct 30 19:31:43.430818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.430748    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh" event={"ID":"0dbdda86-eb09-4f7c-95ab-5e1980209f1b","Type":"ContainerDied","Data":"d36b34ba2478145ebd26c42e41db27bb3e6a154ee3a1d0b9e4979bb5ad4081af"}
Oct 30 19:31:43.432889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.432871    5085 generic.go:356] "Generic (PLEG): container finished" podID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerID="3ffc991bb4b7542b5e324e4cf11054a7a9265498d99409a142e3f14a516565fd" exitCode=0
Oct 30 19:31:43.432994 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:43.432941    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5" event={"ID":"4ebd25e5-f389-43bb-b01f-f98896e872c5","Type":"ContainerDied","Data":"3ffc991bb4b7542b5e324e4cf11054a7a9265498d99409a142e3f14a516565fd"}
Oct 30 19:31:44.444756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:44.443716    5085 generic.go:356] "Generic (PLEG): container finished" podID="17c9d0d6-0003-445d-b928-e38da16560a8" containerID="bb489ed2ce05e360bb7ef1eadbf443bfce6acf04bf708beacc8c55010d13b2d6" exitCode=0
Oct 30 19:31:44.444756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:44.443765    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerDied","Data":"bb489ed2ce05e360bb7ef1eadbf443bfce6acf04bf708beacc8c55010d13b2d6"}
Oct 30 19:31:51.981146 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:31:51.981091    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:31:54.533792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:54.533067    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerStarted","Data":"07f798f94e503d67fa5ed663e1a08174455e26724e065051429b405eb7e6cea6"}
Oct 30 19:31:54.650631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:54.649514    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" podStartSLOduration=2.529682893 podStartE2EDuration="27.649492324s" totalImagesPullingTime="25.119809431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:54.647940613 +0000 UTC m=+424.152344373" watchObservedRunningTime="2026-10-30 19:31:54.649492324 +0000 UTC m=+424.153896073"
Oct 30 19:31:55.546988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:55.546948    5085 generic.go:356] "Generic (PLEG): container finished" podID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerID="07f798f94e503d67fa5ed663e1a08174455e26724e065051429b405eb7e6cea6" exitCode=0
Oct 30 19:31:55.547476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:55.547029    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerDied","Data":"07f798f94e503d67fa5ed663e1a08174455e26724e065051429b405eb7e6cea6"}
Oct 30 19:31:56.786649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.786610    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:56.924997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.924088    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\") pod \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") "
Oct 30 19:31:56.924997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.924156    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\") pod \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") "
Oct 30 19:31:56.924997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.924233    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\" (UniqueName: \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\") pod \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\" (UID: \"6a1d6e7b-b2c4-441a-9261-0d0977544f6f\") "
Oct 30 19:31:56.928032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.927985    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle" pod "6a1d6e7b-b2c4-441a-9261-0d0977544f6f" (UID: "6a1d6e7b-b2c4-441a-9261-0d0977544f6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:56.932601 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.932543    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util" pod "6a1d6e7b-b2c4-441a-9261-0d0977544f6f" (UID: "6a1d6e7b-b2c4-441a-9261-0d0977544f6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:31:56.933000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:56.932961    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72" pod "6a1d6e7b-b2c4-441a-9261-0d0977544f6f" (UID: "6a1d6e7b-b2c4-441a-9261-0d0977544f6f"). InnerVolumeSpecName "kube-api-access-v5c72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:57.025730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.025687    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:57.025730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.025730    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:57.025934 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.025749    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5c72\" (UniqueName: \"kubernetes.io/projected/6a1d6e7b-b2c4-441a-9261-0d0977544f6f-kube-api-access-v5c72\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:31:57.573212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.573173    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242" event={"ID":"6a1d6e7b-b2c4-441a-9261-0d0977544f6f","Type":"ContainerDied","Data":"3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a"}
Oct 30 19:31:57.573403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.573238    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af9e9e5b817e515c98ae3f8ef005d9088d56794342b843f44395cebd979e27a"
Oct 30 19:31:57.573403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:57.573324    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71v5242"
Oct 30 19:31:58.582713 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:58.582220    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerStarted","Data":"e2b2e76ca5bcfe8c984a22d9576837e6ef819e96dedb56243e8dcebdc9108851"}
Oct 30 19:31:58.716836 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:58.716779    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" podStartSLOduration=4.315669094 podStartE2EDuration="36.716764659s" totalImagesPullingTime="32.401095565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:58.71575631 +0000 UTC m=+428.220160056" watchObservedRunningTime="2026-10-30 19:31:58.716764659 +0000 UTC m=+428.221168405"
Oct 30 19:31:59.593601 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.593544    5085 generic.go:356] "Generic (PLEG): container finished" podID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerID="e2b2e76ca5bcfe8c984a22d9576837e6ef819e96dedb56243e8dcebdc9108851" exitCode=0
Oct 30 19:31:59.594117 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.593633    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerDied","Data":"e2b2e76ca5bcfe8c984a22d9576837e6ef819e96dedb56243e8dcebdc9108851"}
Oct 30 19:31:59.595805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.595780    5085 generic.go:356] "Generic (PLEG): container finished" podID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerID="5b231abaa7e7b1ea338e384822e3db67360c46f6c683ab18813197e753b95211" exitCode=0
Oct 30 19:31:59.595946 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.595852    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq" event={"ID":"022fdc57-450c-4bdd-ac98-27ab831eafed","Type":"ContainerDied","Data":"5b231abaa7e7b1ea338e384822e3db67360c46f6c683ab18813197e753b95211"}
Oct 30 19:31:59.597803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.597776    5085 generic.go:356] "Generic (PLEG): container finished" podID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerID="1516e619245cd65fd49b21735cfb872536f5d6544ee43e4b318ea4c0cfba6c2b" exitCode=0
Oct 30 19:31:59.597898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.597815    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh" event={"ID":"a3f96535-08a5-4d56-a73c-59c8d5a18ebf","Type":"ContainerDied","Data":"1516e619245cd65fd49b21735cfb872536f5d6544ee43e4b318ea4c0cfba6c2b"}
Oct 30 19:31:59.599964 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.599943    5085 generic.go:356] "Generic (PLEG): container finished" podID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerID="4795caf934ec5eb6885ecec8c3dba67b4b974f4dbd5cbf40aabd6a52afc3c2bb" exitCode=0
Oct 30 19:31:59.600064 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.600020    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh" event={"ID":"0dbdda86-eb09-4f7c-95ab-5e1980209f1b","Type":"ContainerDied","Data":"4795caf934ec5eb6885ecec8c3dba67b4b974f4dbd5cbf40aabd6a52afc3c2bb"}
Oct 30 19:31:59.602198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.602175    5085 generic.go:356] "Generic (PLEG): container finished" podID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerID="0c4a08d1af67b625619bb2bdf54a112efff56a5c688d6f56ab03866829437e9f" exitCode=0
Oct 30 19:31:59.602300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.602255    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5" event={"ID":"4ebd25e5-f389-43bb-b01f-f98896e872c5","Type":"ContainerDied","Data":"0c4a08d1af67b625619bb2bdf54a112efff56a5c688d6f56ab03866829437e9f"}
Oct 30 19:31:59.604251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:31:59.604220    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerStarted","Data":"c72c4a5e3edd63731d338625e8acd9da491347998b1821da7dc3e9c0151d3504"}
Oct 30 19:32:00.610389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.610353    5085 generic.go:356] "Generic (PLEG): container finished" podID="17c9d0d6-0003-445d-b928-e38da16560a8" containerID="c72c4a5e3edd63731d338625e8acd9da491347998b1821da7dc3e9c0151d3504" exitCode=0
Oct 30 19:32:00.610855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.610433    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerDied","Data":"c72c4a5e3edd63731d338625e8acd9da491347998b1821da7dc3e9c0151d3504"}
Oct 30 19:32:00.758144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.758117    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:32:00.782997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.782957    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"]
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783813    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="extract"
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783837    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="extract"
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783864    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="pull"
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783873    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="pull"
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783907    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="util"
Oct 30 19:32:00.783922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783915    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="util"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783937    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="pull"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783946    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="pull"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783963    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="util"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783971    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="util"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783986    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="extract"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.783994    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="extract"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.784100    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1d6e7b-b2c4-441a-9261-0d0977544f6f" containerName="extract"
Oct 30 19:32:00.784320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.784115    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f96535-08a5-4d56-a73c-59c8d5a18ebf" containerName="extract"
Oct 30 19:32:00.788078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.788059    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.804491 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.804468    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:32:00.832278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.832217    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:32:00.841492 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.841453    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:32:00.841674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.841655    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dvbx8\""
Oct 30 19:32:00.842161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.842140    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 19:32:00.842289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.842271    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:32:00.842354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.842342    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:32:00.842424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.842413    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 19:32:00.848219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.846375    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:32:00.851102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.851075    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:32:00.860900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.860822    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\") pod \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") "
Oct 30 19:32:00.860900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.860887    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\") pod \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") "
Oct 30 19:32:00.861067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.860965    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\" (UniqueName: \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\") pod \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\" (UID: \"a3f96535-08a5-4d56-a73c-59c8d5a18ebf\") "
Oct 30 19:32:00.861625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.861598    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle" pod "a3f96535-08a5-4d56-a73c-59c8d5a18ebf" (UID: "a3f96535-08a5-4d56-a73c-59c8d5a18ebf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.863317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.863283    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7" pod "a3f96535-08a5-4d56-a73c-59c8d5a18ebf" (UID: "a3f96535-08a5-4d56-a73c-59c8d5a18ebf"). InnerVolumeSpecName "kube-api-access-29ck7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.865837 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.865811    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util" pod "a3f96535-08a5-4d56-a73c-59c8d5a18ebf" (UID: "a3f96535-08a5-4d56-a73c-59c8d5a18ebf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.948342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.948306    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"]
Oct 30 19:32:00.961695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961665    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\" (UniqueName: \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\") pod \"4ebd25e5-f389-43bb-b01f-f98896e872c5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") "
Oct 30 19:32:00.961849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961712    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\") pod \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") "
Oct 30 19:32:00.961849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961733    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\") pod \"4ebd25e5-f389-43bb-b01f-f98896e872c5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") "
Oct 30 19:32:00.961849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961774    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\" (UniqueName: \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\") pod \"022fdc57-450c-4bdd-ac98-27ab831eafed\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") "
Oct 30 19:32:00.961849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961806    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\") pod \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") "
Oct 30 19:32:00.961849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961832    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\" (UniqueName: \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\") pod \"d985353a-0b0e-4e03-87dc-7478de4d2ade\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") "
Oct 30 19:32:00.962206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.961982    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\") pod \"d985353a-0b0e-4e03-87dc-7478de4d2ade\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") "
Oct 30 19:32:00.962285 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962243    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\") pod \"4ebd25e5-f389-43bb-b01f-f98896e872c5\" (UID: \"4ebd25e5-f389-43bb-b01f-f98896e872c5\") "
Oct 30 19:32:00.962285 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962280    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\") pod \"d985353a-0b0e-4e03-87dc-7478de4d2ade\" (UID: \"d985353a-0b0e-4e03-87dc-7478de4d2ade\") "
Oct 30 19:32:00.962391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962325    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\") pod \"022fdc57-450c-4bdd-ac98-27ab831eafed\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") "
Oct 30 19:32:00.962391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962356    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\") pod \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\" (UID: \"0dbdda86-eb09-4f7c-95ab-5e1980209f1b\") "
Oct 30 19:32:00.962495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962394    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle" pod "4ebd25e5-f389-43bb-b01f-f98896e872c5" (UID: "4ebd25e5-f389-43bb-b01f-f98896e872c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.962495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962427    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\") pod \"022fdc57-450c-4bdd-ac98-27ab831eafed\" (UID: \"022fdc57-450c-4bdd-ac98-27ab831eafed\") "
Oct 30 19:32:00.962633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962581    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962618    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962648    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l2jl\" (UniqueName: \"kubernetes.io/projected/21281d39-bb45-447d-aa50-051d17103b14-kube-api-access-4l2jl\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962708    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962766    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962775    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle" pod "d985353a-0b0e-4e03-87dc-7478de4d2ade" (UID: "d985353a-0b0e-4e03-87dc-7478de4d2ade"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.962960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962886    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:00.962960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962944    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ck7\" (UniqueName: \"kubernetes.io/projected/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-kube-api-access-29ck7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.963066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962960    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.963066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962974    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.963066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.962988    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a3f96535-08a5-4d56-a73c-59c8d5a18ebf-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.963066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.963000    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:00.964338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.964309    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh" pod "4ebd25e5-f389-43bb-b01f-f98896e872c5" (UID: "4ebd25e5-f389-43bb-b01f-f98896e872c5"). InnerVolumeSpecName "kube-api-access-vjwhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.964463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.964384    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk" pod "022fdc57-450c-4bdd-ac98-27ab831eafed" (UID: "022fdc57-450c-4bdd-ac98-27ab831eafed"). InnerVolumeSpecName "kube-api-access-n7vpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.964885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.964860    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle" pod "0dbdda86-eb09-4f7c-95ab-5e1980209f1b" (UID: "0dbdda86-eb09-4f7c-95ab-5e1980209f1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.964997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.964971    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle" pod "022fdc57-450c-4bdd-ac98-27ab831eafed" (UID: "022fdc57-450c-4bdd-ac98-27ab831eafed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.965754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.965723    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf" pod "d985353a-0b0e-4e03-87dc-7478de4d2ade" (UID: "d985353a-0b0e-4e03-87dc-7478de4d2ade"). InnerVolumeSpecName "kube-api-access-2k9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.966538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.966502    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm" pod "0dbdda86-eb09-4f7c-95ab-5e1980209f1b" (UID: "0dbdda86-eb09-4f7c-95ab-5e1980209f1b"). InnerVolumeSpecName "kube-api-access-56rsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:00.968298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.968271    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util" pod "4ebd25e5-f389-43bb-b01f-f98896e872c5" (UID: "4ebd25e5-f389-43bb-b01f-f98896e872c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.970442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.970417    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util" pod "0dbdda86-eb09-4f7c-95ab-5e1980209f1b" (UID: "0dbdda86-eb09-4f7c-95ab-5e1980209f1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.970657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.970634    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util" pod "d985353a-0b0e-4e03-87dc-7478de4d2ade" (UID: "d985353a-0b0e-4e03-87dc-7478de4d2ade"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:00.971406 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:00.971388    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util" pod "022fdc57-450c-4bdd-ac98-27ab831eafed" (UID: "022fdc57-450c-4bdd-ac98-27ab831eafed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:01.064025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.063989    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064036    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064066    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064092    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l2jl\" (UniqueName: \"kubernetes.io/projected/21281d39-bb45-447d-aa50-051d17103b14-kube-api-access-4l2jl\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064275    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064341    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.064481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064402    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064451    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjwhh\" (UniqueName: \"kubernetes.io/projected/4ebd25e5-f389-43bb-b01f-f98896e872c5-kube-api-access-vjwhh\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064475    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064493    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7vpk\" (UniqueName: \"kubernetes.io/projected/022fdc57-450c-4bdd-ac98-27ab831eafed-kube-api-access-n7vpk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064508    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rsm\" (UniqueName: \"kubernetes.io/projected/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-kube-api-access-56rsm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064525    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9zf\" (UniqueName: \"kubernetes.io/projected/d985353a-0b0e-4e03-87dc-7478de4d2ade-kube-api-access-2k9zf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064539    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d985353a-0b0e-4e03-87dc-7478de4d2ade-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064549    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ebd25e5-f389-43bb-b01f-f98896e872c5-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064556    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/022fdc57-450c-4bdd-ac98-27ab831eafed-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064585    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0dbdda86-eb09-4f7c-95ab-5e1980209f1b-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.064670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.064621    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-manifests\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.066409 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.066379    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21281d39-bb45-447d-aa50-051d17103b14-tmp\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.066665 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.066646    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.066708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.066678    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-metrics-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.067016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.066997    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/21281d39-bb45-447d-aa50-051d17103b14-webhook-cert\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.130040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.129973    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l2jl\" (UniqueName: \"kubernetes.io/projected/21281d39-bb45-447d-aa50-051d17103b14-kube-api-access-4l2jl\") pod \"opendatahub-operator-controller-manager-f84ddd5d6-2cmxc\" (UID: \"21281d39-bb45-447d-aa50-051d17103b14\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.416469 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.416386    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:01.616328 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.616299    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5"
Oct 30 19:32:01.616328 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.616305    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c66l5s5" event={"ID":"4ebd25e5-f389-43bb-b01f-f98896e872c5","Type":"ContainerDied","Data":"dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0"}
Oct 30 19:32:01.616868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.616353    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dafa83cdc7e6b0fbf095738a3ec86b80fd3eb59aae9dc1abc718db797e87d3a0"
Oct 30 19:32:01.618408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.618388    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc"
Oct 30 19:32:01.618408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.618398    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6442m6lc" event={"ID":"d985353a-0b0e-4e03-87dc-7478de4d2ade","Type":"ContainerDied","Data":"758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2"}
Oct 30 19:32:01.618611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.618433    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758824d0ee84bd4723912076a00afac3ebd4d0fc2f813bff2c04db7773999bd2"
Oct 30 19:32:01.620408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.620392    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq"
Oct 30 19:32:01.620501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.620406    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cr6zxq" event={"ID":"022fdc57-450c-4bdd-ac98-27ab831eafed","Type":"ContainerDied","Data":"ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409"}
Oct 30 19:32:01.620501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.620443    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6136214b725f6e134ace7735aa349c967cbd89775a7d73ff42ec5e6c0bc409"
Oct 30 19:32:01.622396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.622377    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh"
Oct 30 19:32:01.622505 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.622393    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccxpwh" event={"ID":"a3f96535-08a5-4d56-a73c-59c8d5a18ebf","Type":"ContainerDied","Data":"8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992"}
Oct 30 19:32:01.622505 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.622439    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6df93dd63d9ce05588f04b3e00c206d32b445329e21551ad7b599fadbf1992"
Oct 30 19:32:01.626798 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.626775    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh" event={"ID":"0dbdda86-eb09-4f7c-95ab-5e1980209f1b","Type":"ContainerDied","Data":"777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d"}
Oct 30 19:32:01.626976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.626952    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="777c84194ee6db225f8c68f064b02e0ba0d023eeba61dfd248ae41570f1aa78d"
Oct 30 19:32:01.626976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.626815    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7942lh"
Oct 30 19:32:01.765473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.765450    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:32:01.871329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.871295    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\") pod \"17c9d0d6-0003-445d-b928-e38da16560a8\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") "
Oct 30 19:32:01.871517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.871338    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\" (UniqueName: \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\") pod \"17c9d0d6-0003-445d-b928-e38da16560a8\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") "
Oct 30 19:32:01.871517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.871420    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\") pod \"17c9d0d6-0003-445d-b928-e38da16560a8\" (UID: \"17c9d0d6-0003-445d-b928-e38da16560a8\") "
Oct 30 19:32:01.872090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.872063    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle" pod "17c9d0d6-0003-445d-b928-e38da16560a8" (UID: "17c9d0d6-0003-445d-b928-e38da16560a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:01.873445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.873421    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk" pod "17c9d0d6-0003-445d-b928-e38da16560a8" (UID: "17c9d0d6-0003-445d-b928-e38da16560a8"). InnerVolumeSpecName "kube-api-access-wzvnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:01.876970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.876944    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util" pod "17c9d0d6-0003-445d-b928-e38da16560a8" (UID: "17c9d0d6-0003-445d-b928-e38da16560a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:32:01.906936 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.906902    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"]
Oct 30 19:32:01.910775 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:01.910750    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21281d39_bb45_447d_aa50_051d17103b14.slice/crio-8cf59fa21512e5841da3b8c5c75caf701dba8af2e9b8e892e6a92aa8b5e89e42 WatchSource:0}: Error finding container 8cf59fa21512e5841da3b8c5c75caf701dba8af2e9b8e892e6a92aa8b5e89e42: Status 404 returned error can't find the container with id 8cf59fa21512e5841da3b8c5c75caf701dba8af2e9b8e892e6a92aa8b5e89e42
Oct 30 19:32:01.972408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.972332    5085 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-util\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.972408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.972357    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzvnk\" (UniqueName: \"kubernetes.io/projected/17c9d0d6-0003-445d-b928-e38da16560a8-kube-api-access-wzvnk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:01.972408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:01.972369    5085 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17c9d0d6-0003-445d-b928-e38da16560a8-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:02.636228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:02.636200    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f"
Oct 30 19:32:02.636686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:02.636196    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6mk8f" event={"ID":"17c9d0d6-0003-445d-b928-e38da16560a8","Type":"ContainerDied","Data":"aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8"}
Oct 30 19:32:02.636686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:02.636319    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4680319e581ed446884ebda9d20c690fa1a53a68a06161bbe3dca2f968e3d8"
Oct 30 19:32:02.637650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:02.637622    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc" event={"ID":"21281d39-bb45-447d-aa50-051d17103b14","Type":"ContainerStarted","Data":"8cf59fa21512e5841da3b8c5c75caf701dba8af2e9b8e892e6a92aa8b5e89e42"}
Oct 30 19:32:06.659384 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:06.659298    5085 generic.go:356] "Generic (PLEG): container finished" podID="21281d39-bb45-447d-aa50-051d17103b14" containerID="54dfb200bc5a5d6389b3af5b895718c4592aa8788cc58d5fc258603b4f1993ad" exitCode=0
Oct 30 19:32:06.659384 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:06.659374    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc" event={"ID":"21281d39-bb45-447d-aa50-051d17103b14","Type":"ContainerDied","Data":"54dfb200bc5a5d6389b3af5b895718c4592aa8788cc58d5fc258603b4f1993ad"}
Oct 30 19:32:07.666974 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:07.666936    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc" event={"ID":"21281d39-bb45-447d-aa50-051d17103b14","Type":"ContainerStarted","Data":"3cabae4ebc4720e348ecbbe32c41e06a0b3abcfc18c8b6aa8d2e6a4ba53484d8"}
Oct 30 19:32:07.667383 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:07.667040    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:07.805098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:07.804989    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc" podStartSLOduration=2.787700851 podStartE2EDuration="7.804974094s" totalImagesPullingTime="4.56895531s" totalInitContainerRuntime="448.317933ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:07.802984856 +0000 UTC m=+437.307388594" watchObservedRunningTime="2026-10-30 19:32:07.804974094 +0000 UTC m=+437.309377838"
Oct 30 19:32:09.925965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.925930    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"]
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926254    5085 cpu_manager.go:414] "containerMap: removing container" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926265    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926276    5085 cpu_manager.go:414] "containerMap: removing container" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="pull"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926284    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="pull"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926301    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926306    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926313    5085 cpu_manager.go:414] "containerMap: removing container" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926318    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="util"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926324    5085 cpu_manager.go:414] "containerMap: removing container" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="pull"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926328    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="pull"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926333    5085 cpu_manager.go:414] "containerMap: removing container" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="extract"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926338    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="extract"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926345    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="pull"
Oct 30 19:32:09.926345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926349    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="pull"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926357    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926365    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926374    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="util"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926378    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="util"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926384    5085 cpu_manager.go:414] "containerMap: removing container" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926388    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926398    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926402    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926411    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="pull"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926416    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="pull"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926422    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926426    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926435    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="util"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926442    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="util"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926455    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="pull"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926460    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="pull"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926517    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="022fdc57-450c-4bdd-ac98-27ab831eafed" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926528    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dbdda86-eb09-4f7c-95ab-5e1980209f1b" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926534    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9d0d6-0003-445d-b928-e38da16560a8" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926541    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebd25e5-f389-43bb-b01f-f98896e872c5" containerName="extract"
Oct 30 19:32:09.926799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.926547    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d985353a-0b0e-4e03-87dc-7478de4d2ade" containerName="extract"
Oct 30 19:32:09.928877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.928861    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:09.941481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.941460    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67j64\" (UniqueName: \"kubernetes.io/projected/5097f139-50b9-40aa-a047-8d94934d00c1-kube-api-access-67j64\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:09.941615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.941501    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5097f139-50b9-40aa-a047-8d94934d00c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:09.979671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:09.979649    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 30 19:32:10.026163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.026135    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:32:10.026643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.026623    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hb6lb\""
Oct 30 19:32:10.028270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.028251    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"]
Oct 30 19:32:10.041884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.041855    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5097f139-50b9-40aa-a047-8d94934d00c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:10.041997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.041952    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67j64\" (UniqueName: \"kubernetes.io/projected/5097f139-50b9-40aa-a047-8d94934d00c1-kube-api-access-67j64\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:10.042230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.042208    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5097f139-50b9-40aa-a047-8d94934d00c1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:10.103052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.103022    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67j64\" (UniqueName: \"kubernetes.io/projected/5097f139-50b9-40aa-a047-8d94934d00c1-kube-api-access-67j64\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-mt68k\" (UID: \"5097f139-50b9-40aa-a047-8d94934d00c1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:10.240112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.240019    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"
Oct 30 19:32:10.708407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:10.707259    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k"]
Oct 30 19:32:11.685720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:11.685682    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k" event={"ID":"5097f139-50b9-40aa-a047-8d94934d00c1","Type":"ContainerStarted","Data":"1cda92263e4be557807609c4ffae18230b3cb8facff47583ad9bb5ea7c0dc1d3"}
Oct 30 19:32:13.698355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:13.698317    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k" event={"ID":"5097f139-50b9-40aa-a047-8d94934d00c1","Type":"ContainerStarted","Data":"853ef42edc964a9147abecbf617e3d2a719d2809b706565421c5930659fe4257"}
Oct 30 19:32:13.905632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:13.905513    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-mt68k" podStartSLOduration=2.078505869 podStartE2EDuration="4.905497378s" totalImagesPullingTime="2.826991509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:13.858577617 +0000 UTC m=+443.362981354" watchObservedRunningTime="2026-10-30 19:32:13.905497378 +0000 UTC m=+443.409901125"
Oct 30 19:32:18.673883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:18.673848    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f84ddd5d6-2cmxc"
Oct 30 19:32:19.319050 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.319016    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"]
Oct 30 19:32:19.321871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.321850    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:19.371362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.371326    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-g5tmk\""
Oct 30 19:32:19.371596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.371533    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:32:19.371690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.371651    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:32:19.413003 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.412972    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"]
Oct 30 19:32:19.421342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.421312    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjc9\" (UniqueName: \"kubernetes.io/projected/bafa56a2-5852-481e-bed8-94df375b14b6-kube-api-access-dtjc9\") pod \"authorino-operator-7b7bb4bc77-5r6fq\" (UID: \"bafa56a2-5852-481e-bed8-94df375b14b6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:19.522411 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.522372    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjc9\" (UniqueName: \"kubernetes.io/projected/bafa56a2-5852-481e-bed8-94df375b14b6-kube-api-access-dtjc9\") pod \"authorino-operator-7b7bb4bc77-5r6fq\" (UID: \"bafa56a2-5852-481e-bed8-94df375b14b6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:19.581090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.581013    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjc9\" (UniqueName: \"kubernetes.io/projected/bafa56a2-5852-481e-bed8-94df375b14b6-kube-api-access-dtjc9\") pod \"authorino-operator-7b7bb4bc77-5r6fq\" (UID: \"bafa56a2-5852-481e-bed8-94df375b14b6\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:19.633122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.633090    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:19.999012 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:19.998980    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"]
Oct 30 19:32:20.002202 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:20.002170    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafa56a2_5852_481e_bed8_94df375b14b6.slice/crio-9da47f8c6f533fa22a3308def5c482083b0861e184464b115470d79e4b37f5af WatchSource:0}: Error finding container 9da47f8c6f533fa22a3308def5c482083b0861e184464b115470d79e4b37f5af: Status 404 returned error can't find the container with id 9da47f8c6f533fa22a3308def5c482083b0861e184464b115470d79e4b37f5af
Oct 30 19:32:20.732416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:20.732377    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq" event={"ID":"bafa56a2-5852-481e-bed8-94df375b14b6","Type":"ContainerStarted","Data":"9da47f8c6f533fa22a3308def5c482083b0861e184464b115470d79e4b37f5af"}
Oct 30 19:32:21.946144 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:32:21.946110    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:32:23.468764 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.468722    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"]
Oct 30 19:32:23.471684 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.471660    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:23.559089 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.559065    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbxc\" (UniqueName: \"kubernetes.io/projected/497a76db-4f65-47d2-9d8d-857e278887dc-kube-api-access-kxbxc\") pod \"limitador-operator-controller-manager-5778bb4449-jt8mk\" (UID: \"497a76db-4f65-47d2-9d8d-857e278887dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:23.568138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.568113    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xpkrd\""
Oct 30 19:32:23.570118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.570091    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"]
Oct 30 19:32:23.659742 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.659705    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbxc\" (UniqueName: \"kubernetes.io/projected/497a76db-4f65-47d2-9d8d-857e278887dc-kube-api-access-kxbxc\") pod \"limitador-operator-controller-manager-5778bb4449-jt8mk\" (UID: \"497a76db-4f65-47d2-9d8d-857e278887dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:23.719899 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.719828    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbxc\" (UniqueName: \"kubernetes.io/projected/497a76db-4f65-47d2-9d8d-857e278887dc-kube-api-access-kxbxc\") pod \"limitador-operator-controller-manager-5778bb4449-jt8mk\" (UID: \"497a76db-4f65-47d2-9d8d-857e278887dc\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:23.748035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.747990    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq" event={"ID":"bafa56a2-5852-481e-bed8-94df375b14b6","Type":"ContainerStarted","Data":"4bc0adcf5035fc6faa46875fd22c19574dfcb3d703fd8d0f19e579c6b2ab1fc1"}
Oct 30 19:32:23.748260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.748239    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:23.787450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.787422    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:23.862312 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:23.862261    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq" podStartSLOduration=1.315043781 podStartE2EDuration="4.862245238s" totalImagesPullingTime="3.547201457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:23.860961781 +0000 UTC m=+453.365365527" watchObservedRunningTime="2026-10-30 19:32:23.862245238 +0000 UTC m=+453.366648986"
Oct 30 19:32:24.156230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:24.156185    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"]
Oct 30 19:32:24.158701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:24.158669    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497a76db_4f65_47d2_9d8d_857e278887dc.slice/crio-8a7dfefbe2ff6b1737fde5caa5b8d7b88ef09c22581b18bf6b2c9250d06e17dd WatchSource:0}: Error finding container 8a7dfefbe2ff6b1737fde5caa5b8d7b88ef09c22581b18bf6b2c9250d06e17dd: Status 404 returned error can't find the container with id 8a7dfefbe2ff6b1737fde5caa5b8d7b88ef09c22581b18bf6b2c9250d06e17dd
Oct 30 19:32:24.754399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:24.754361    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk" event={"ID":"497a76db-4f65-47d2-9d8d-857e278887dc","Type":"ContainerStarted","Data":"8a7dfefbe2ff6b1737fde5caa5b8d7b88ef09c22581b18bf6b2c9250d06e17dd"}
Oct 30 19:32:26.767011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:26.766973    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk" event={"ID":"497a76db-4f65-47d2-9d8d-857e278887dc","Type":"ContainerStarted","Data":"22cc5751dbb89db6bfc4ef6e87653d7a5e268c5daa8bf4686c9fecaeafabfbf1"}
Oct 30 19:32:26.767462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:26.767161    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:26.879547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:26.879449    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk" podStartSLOduration=1.4359834839999999 podStartE2EDuration="3.879434247s" totalImagesPullingTime="2.443450763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:26.877319605 +0000 UTC m=+456.381723354" watchObservedRunningTime="2026-10-30 19:32:26.879434247 +0000 UTC m=+456.383837973"
Oct 30 19:32:27.341897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.341860    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"]
Oct 30 19:32:27.344544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.344527    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:27.394308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.394270    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gkfs\" (UniqueName: \"kubernetes.io/projected/eeb9cbac-453c-440e-9608-2b80fb7424d4-kube-api-access-6gkfs\") pod \"dns-operator-controller-manager-8557cbdb84-gz5b9\" (UID: \"eeb9cbac-453c-440e-9608-2b80fb7424d4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:27.394668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.394648    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 19:32:27.399869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.399848    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-478xh\""
Oct 30 19:32:27.443238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.443206    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"]
Oct 30 19:32:27.495851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.495820    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gkfs\" (UniqueName: \"kubernetes.io/projected/eeb9cbac-453c-440e-9608-2b80fb7424d4-kube-api-access-6gkfs\") pod \"dns-operator-controller-manager-8557cbdb84-gz5b9\" (UID: \"eeb9cbac-453c-440e-9608-2b80fb7424d4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:27.554534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.554503    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gkfs\" (UniqueName: \"kubernetes.io/projected/eeb9cbac-453c-440e-9608-2b80fb7424d4-kube-api-access-6gkfs\") pod \"dns-operator-controller-manager-8557cbdb84-gz5b9\" (UID: \"eeb9cbac-453c-440e-9608-2b80fb7424d4\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:27.656442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:27.656367    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:28.020928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:28.020896    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"]
Oct 30 19:32:28.025197 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:28.025165    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb9cbac_453c_440e_9608_2b80fb7424d4.slice/crio-57d1ef93292e8f8f7589b9d5dde4dc78f643bce9303fd2056837d2a5131f32f5 WatchSource:0}: Error finding container 57d1ef93292e8f8f7589b9d5dde4dc78f643bce9303fd2056837d2a5131f32f5: Status 404 returned error can't find the container with id 57d1ef93292e8f8f7589b9d5dde4dc78f643bce9303fd2056837d2a5131f32f5
Oct 30 19:32:28.778585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:28.778535    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9" event={"ID":"eeb9cbac-453c-440e-9608-2b80fb7424d4","Type":"ContainerStarted","Data":"57d1ef93292e8f8f7589b9d5dde4dc78f643bce9303fd2056837d2a5131f32f5"}
Oct 30 19:32:31.794329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:31.794290    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9" event={"ID":"eeb9cbac-453c-440e-9608-2b80fb7424d4","Type":"ContainerStarted","Data":"4f271b05dce79e33ca107a4e5e7a44e0432a9d8405e7dda55639b05ad18e376b"}
Oct 30 19:32:31.794860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:31.794421    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:31.913680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:31.913624    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9" podStartSLOduration=2.144017126 podStartE2EDuration="4.913609501s" totalImagesPullingTime="2.769592375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:31.912292424 +0000 UTC m=+461.416696170" watchObservedRunningTime="2026-10-30 19:32:31.913609501 +0000 UTC m=+461.418013247"
Oct 30 19:32:33.952159 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:33.952123    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:32:33.956210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:33.956187    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.006065 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.006037    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:32:34.050201 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.050161    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.050393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.050212    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvvj\" (UniqueName: \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.050472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.050436    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gmtfr\""
Oct 30 19:32:34.151253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.151215    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvvj\" (UniqueName: \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.151423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.151347    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.151779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.151760    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.209794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.209722    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvvj\" (UniqueName: \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\") pod \"kuadrant-operator-controller-manager-cf8469d7b-vjj8v\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.268526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.268494    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:34.684275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.684251    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:32:34.686812 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:34.686777    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a006a13_cb81_4f70_b3c7_53f7dc7a5dbc.slice/crio-af421582fcb7985c1bb4730821186e85d475614a0585b4d6636e7c8c61d4e1ed WatchSource:0}: Error finding container af421582fcb7985c1bb4730821186e85d475614a0585b4d6636e7c8c61d4e1ed: Status 404 returned error can't find the container with id af421582fcb7985c1bb4730821186e85d475614a0585b4d6636e7c8c61d4e1ed
Oct 30 19:32:34.756777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.756749    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5r6fq"
Oct 30 19:32:34.809658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:34.809626    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" event={"ID":"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc","Type":"ContainerStarted","Data":"af421582fcb7985c1bb4730821186e85d475614a0585b4d6636e7c8c61d4e1ed"}
Oct 30 19:32:37.774486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:37.774453    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jt8mk"
Oct 30 19:32:40.842393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:40.842348    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" event={"ID":"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc","Type":"ContainerStarted","Data":"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"}
Oct 30 19:32:40.842824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:40.842415    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:41.054807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.054751    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" podStartSLOduration=2.764694821 podStartE2EDuration="8.05473228s" totalImagesPullingTime="5.290037459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:41.006317548 +0000 UTC m=+470.510721292" watchObservedRunningTime="2026-10-30 19:32:41.05473228 +0000 UTC m=+470.559136027"
Oct 30 19:32:41.492886 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.492851    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"]
Oct 30 19:32:41.495416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.495398    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.548499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.548469    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qmlsb\""
Oct 30 19:32:41.556067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.556022    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"]
Oct 30 19:32:41.619432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619394    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5d6579-23f1-4728-8af6-8bdf452d8e96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619439    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn8fj\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-kube-api-access-vn8fj\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619530    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619573    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619663    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619717    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619775    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619806    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.619889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.619851    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.720941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.720907    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.720941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.720941    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721152 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.720961    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721152 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721126    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721181    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5d6579-23f1-4728-8af6-8bdf452d8e96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721224    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn8fj\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-kube-api-access-vn8fj\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721258    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721288    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721361    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721364    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721639 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721449    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-data\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721709    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721833    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.721982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.721966    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df5d6579-23f1-4728-8af6-8bdf452d8e96-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.723506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.723483    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.723790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.723771    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.783280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.783211    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-istio-token\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.783413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.783317    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn8fj\" (UniqueName: \"kubernetes.io/projected/df5d6579-23f1-4728-8af6-8bdf452d8e96-kube-api-access-vn8fj\") pod \"data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt\" (UID: \"df5d6579-23f1-4728-8af6-8bdf452d8e96\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:41.807260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:41.807230    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:42.226014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.225987    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"]
Oct 30 19:32:42.228255 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:42.228217    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5d6579_23f1_4728_8af6_8bdf452d8e96.slice/crio-f61d2f655465fcf74e6502f71a12b6f7180f089ea5ad6cf723e0f65a859eb9d4 WatchSource:0}: Error finding container f61d2f655465fcf74e6502f71a12b6f7180f089ea5ad6cf723e0f65a859eb9d4: Status 404 returned error can't find the container with id f61d2f655465fcf74e6502f71a12b6f7180f089ea5ad6cf723e0f65a859eb9d4
Oct 30 19:32:42.230441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.230410    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:32:42.230490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.230475    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:32:42.230529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.230514    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:32:42.801659 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.801630    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-gz5b9"
Oct 30 19:32:42.854314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.854278    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt" event={"ID":"df5d6579-23f1-4728-8af6-8bdf452d8e96","Type":"ContainerStarted","Data":"ee3c0dd2713cf92eab8bcafc1ac7ad5acba175e532067e65a1dc55f74c61cf95"}
Oct 30 19:32:42.854484 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:42.854343    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt" event={"ID":"df5d6579-23f1-4728-8af6-8bdf452d8e96","Type":"ContainerStarted","Data":"f61d2f655465fcf74e6502f71a12b6f7180f089ea5ad6cf723e0f65a859eb9d4"}
Oct 30 19:32:43.033745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:43.033690    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt" podStartSLOduration=2.033674046 podStartE2EDuration="2.033674046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:43.031631209 +0000 UTC m=+472.536034956" watchObservedRunningTime="2026-10-30 19:32:43.033674046 +0000 UTC m=+472.538077792"
Oct 30 19:32:43.808119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:43.808079    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:43.812825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:43.812802    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:43.858832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:43.858795    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:43.859586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:43.859550    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt"
Oct 30 19:32:46.941761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:46.941720    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:46.945674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:46.945650    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:46.998839 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:46.998803    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:47.043184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.043155    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-29dpz\""
Oct 30 19:32:47.074612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.074557    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbrsf\" (UniqueName: \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\") pod \"authorino-57f7f8659f-b8z4n\" (UID: \"02b72ce3-a4b3-4920-bab8-e3ce8b65322a\") " pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:47.175917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.175874    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbrsf\" (UniqueName: \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\") pod \"authorino-57f7f8659f-b8z4n\" (UID: \"02b72ce3-a4b3-4920-bab8-e3ce8b65322a\") " pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:47.235939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.235860    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbrsf\" (UniqueName: \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\") pod \"authorino-57f7f8659f-b8z4n\" (UID: \"02b72ce3-a4b3-4920-bab8-e3ce8b65322a\") " pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:47.270509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.270476    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:47.635323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.635298    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:47.637254 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:47.637220    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02b72ce3_a4b3_4920_bab8_e3ce8b65322a.slice/crio-b8385e96d0c34238a45f38199bd99d22c867a7485a597420808d360a8735d006 WatchSource:0}: Error finding container b8385e96d0c34238a45f38199bd99d22c867a7485a597420808d360a8735d006: Status 404 returned error can't find the container with id b8385e96d0c34238a45f38199bd99d22c867a7485a597420808d360a8735d006
Oct 30 19:32:47.884961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:47.884882    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" event={"ID":"02b72ce3-a4b3-4920-bab8-e3ce8b65322a","Type":"ContainerStarted","Data":"b8385e96d0c34238a45f38199bd99d22c867a7485a597420808d360a8735d006"}
Oct 30 19:32:50.139360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.139329    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:32:50.143995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.143971    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:32:50.144126 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.144097    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:32:50.148030 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.148010    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.195555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.195521    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:32:50.239807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.239779    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:32:50.239807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.239779    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rq9w4\""
Oct 30 19:32:50.283969 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.283918    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:32:50.302638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.302606    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwfw\" (UniqueName: \"kubernetes.io/projected/46f9d21e-6c67-44c5-a817-32f250962178-kube-api-access-gxwfw\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.302792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.302645    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46f9d21e-6c67-44c5-a817-32f250962178-config-file\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.302837 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.302792    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mspq8\" (UniqueName: \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\") pod \"authorino-56c6c9ddd5-fstjg\" (UID: \"d60224c2-498c-4ce3-a3df-1ecce93b0ed6\") " pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:32:50.403462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.403390    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwfw\" (UniqueName: \"kubernetes.io/projected/46f9d21e-6c67-44c5-a817-32f250962178-kube-api-access-gxwfw\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.403462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.403429    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46f9d21e-6c67-44c5-a817-32f250962178-config-file\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.403672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.403506    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mspq8\" (UniqueName: \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\") pod \"authorino-56c6c9ddd5-fstjg\" (UID: \"d60224c2-498c-4ce3-a3df-1ecce93b0ed6\") " pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:32:50.404235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.404214    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46f9d21e-6c67-44c5-a817-32f250962178-config-file\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.463829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.463787    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mspq8\" (UniqueName: \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\") pod \"authorino-56c6c9ddd5-fstjg\" (UID: \"d60224c2-498c-4ce3-a3df-1ecce93b0ed6\") " pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:32:50.464193 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.464170    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwfw\" (UniqueName: \"kubernetes.io/projected/46f9d21e-6c67-44c5-a817-32f250962178-kube-api-access-gxwfw\") pod \"limitador-limitador-7c9b87d574-mfq8q\" (UID: \"46f9d21e-6c67-44c5-a817-32f250962178\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.465717 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.465700    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:32:50.522944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.522902    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:32:50.546407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.546380    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:50.890914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.890875    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:32:50.901764 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.901733    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" event={"ID":"02b72ce3-a4b3-4920-bab8-e3ce8b65322a","Type":"ContainerStarted","Data":"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"}
Oct 30 19:32:50.928115 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:50.928089    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd60224c2_498c_4ce3_a3df_1ecce93b0ed6.slice/crio-0b3cc54d2a880a9cd5be5277019e2280dc3babbb3eabfaea5e4d42c2b43a68a6 WatchSource:0}: Error finding container 0b3cc54d2a880a9cd5be5277019e2280dc3babbb3eabfaea5e4d42c2b43a68a6: Status 404 returned error can't find the container with id 0b3cc54d2a880a9cd5be5277019e2280dc3babbb3eabfaea5e4d42c2b43a68a6
Oct 30 19:32:50.940199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:50.940163    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:32:50.941979 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:32:50.941957    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f9d21e_6c67_44c5_a817_32f250962178.slice/crio-8327d8701b338b3d88ae2807144b530bffbe10613b2f9d4dac21cf15b22b3160 WatchSource:0}: Error finding container 8327d8701b338b3d88ae2807144b530bffbe10613b2f9d4dac21cf15b22b3160: Status 404 returned error can't find the container with id 8327d8701b338b3d88ae2807144b530bffbe10613b2f9d4dac21cf15b22b3160
Oct 30 19:32:51.011115 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:51.011066    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" podStartSLOduration=2.518913755 podStartE2EDuration="5.011050516s" totalImagesPullingTime="2.492136761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:51.010671 +0000 UTC m=+480.515074751" watchObservedRunningTime="2026-10-30 19:32:51.011050516 +0000 UTC m=+480.515454266"
Oct 30 19:32:51.851260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:51.851227    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:32:51.908332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:51.908294    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" event={"ID":"d60224c2-498c-4ce3-a3df-1ecce93b0ed6","Type":"ContainerStarted","Data":"0b3cc54d2a880a9cd5be5277019e2280dc3babbb3eabfaea5e4d42c2b43a68a6"}
Oct 30 19:32:51.909620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:51.909583    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q" event={"ID":"46f9d21e-6c67-44c5-a817-32f250962178","Type":"ContainerStarted","Data":"8327d8701b338b3d88ae2807144b530bffbe10613b2f9d4dac21cf15b22b3160"}
Oct 30 19:32:51.937598 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:32:51.937557    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:32:52.917741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:52.917702    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" event={"ID":"d60224c2-498c-4ce3-a3df-1ecce93b0ed6","Type":"ContainerStarted","Data":"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"}
Oct 30 19:32:53.026819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.026771    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" podStartSLOduration=1.7154491539999999 podStartE2EDuration="3.026758522s" totalImagesPullingTime="1.311309368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:53.02477964 +0000 UTC m=+482.529183388" watchObservedRunningTime="2026-10-30 19:32:53.026758522 +0000 UTC m=+482.531162296"
Oct 30 19:32:53.057047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.057024    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:53.057199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.057174    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" podUID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" containerName="authorino" containerID="cri-o://00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262" gracePeriod=30
Oct 30 19:32:53.307264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.307238    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:53.427429 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.427335    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\" (UniqueName: \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\") pod \"02b72ce3-a4b3-4920-bab8-e3ce8b65322a\" (UID: \"02b72ce3-a4b3-4920-bab8-e3ce8b65322a\") "
Oct 30 19:32:53.429467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.429437    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf" pod "02b72ce3-a4b3-4920-bab8-e3ce8b65322a" (UID: "02b72ce3-a4b3-4920-bab8-e3ce8b65322a"). InnerVolumeSpecName "kube-api-access-gbrsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:32:53.529082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.529041    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbrsf\" (UniqueName: \"kubernetes.io/projected/02b72ce3-a4b3-4920-bab8-e3ce8b65322a-kube-api-access-gbrsf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:32:53.924915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.924882    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q" event={"ID":"46f9d21e-6c67-44c5-a817-32f250962178","Type":"ContainerStarted","Data":"3fd0cc2c9ee0b7711eb9743a13648bb8da6927f5a28934a700e70c242516b535"}
Oct 30 19:32:53.925307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.924979    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:32:53.926133 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.926108    5085 generic.go:356] "Generic (PLEG): container finished" podID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" containerID="00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262" exitCode=0
Oct 30 19:32:53.926231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.926163    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-b8z4n"
Oct 30 19:32:53.926231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.926186    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" event={"ID":"02b72ce3-a4b3-4920-bab8-e3ce8b65322a","Type":"ContainerDied","Data":"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"}
Oct 30 19:32:53.926231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.926225    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-b8z4n" event={"ID":"02b72ce3-a4b3-4920-bab8-e3ce8b65322a","Type":"ContainerDied","Data":"b8385e96d0c34238a45f38199bd99d22c867a7485a597420808d360a8735d006"}
Oct 30 19:32:53.926329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.926241    5085 scope.go:122] "RemoveContainer" containerID="00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"
Oct 30 19:32:53.935119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.935103    5085 scope.go:122] "RemoveContainer" containerID="00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"
Oct 30 19:32:53.935383 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:32:53.935361    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262\": container with ID starting with 00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262 not found: ID does not exist" containerID="00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"
Oct 30 19:32:53.935442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:53.935396    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262"} err="failed to get container status \"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262\": rpc error: code = NotFound desc = could not find container \"00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262\": container with ID starting with 00e03c49424917872b1eff93a06424ec80ba74d3df4b9b984f74e718a3ad4262 not found: ID does not exist"
Oct 30 19:32:54.040322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:54.040266    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q" podStartSLOduration=1.988196595 podStartE2EDuration="4.040252259s" totalImagesPullingTime="2.052055664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:54.038005433 +0000 UTC m=+483.542409180" watchObservedRunningTime="2026-10-30 19:32:54.040252259 +0000 UTC m=+483.544656005"
Oct 30 19:32:54.147523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:54.147489    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:54.156056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:54.156027    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-b8z4n"]
Oct 30 19:32:54.629733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:32:54.629704    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" path="/var/lib/kubelet/pods/02b72ce3-a4b3-4920-bab8-e3ce8b65322a/volumes"
Oct 30 19:33:04.934116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:04.934078    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"
Oct 30 19:33:09.649656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.649616    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:33:09.650136 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.649918    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" podUID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" containerName="manager" containerID="cri-o://3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b" gracePeriod=10
Oct 30 19:33:09.899894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.899831    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:33:09.953857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.953822    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\") pod \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") "
Oct 30 19:33:09.954031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.953901    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\" (UniqueName: \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\") pod \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\" (UID: \"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc\") "
Oct 30 19:33:09.954240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.954220    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume" pod "1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" (UID: "1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:09.956024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:09.955999    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj" pod "1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" (UID: "1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc"). InnerVolumeSpecName "kube-api-access-frvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:10.008371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.008339    5085 generic.go:356] "Generic (PLEG): container finished" podID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" containerID="3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b" exitCode=0
Oct 30 19:33:10.008526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.008393    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"
Oct 30 19:33:10.008526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.008415    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" event={"ID":"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc","Type":"ContainerDied","Data":"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"}
Oct 30 19:33:10.008526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.008468    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v" event={"ID":"1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc","Type":"ContainerDied","Data":"af421582fcb7985c1bb4730821186e85d475614a0585b4d6636e7c8c61d4e1ed"}
Oct 30 19:33:10.008526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.008489    5085 scope.go:122] "RemoveContainer" containerID="3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"
Oct 30 19:33:10.017513 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.017494    5085 scope.go:122] "RemoveContainer" containerID="3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"
Oct 30 19:33:10.017780 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:10.017764    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b\": container with ID starting with 3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b not found: ID does not exist" containerID="3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"
Oct 30 19:33:10.017849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.017788    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b"} err="failed to get container status \"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b\": rpc error: code = NotFound desc = could not find container \"3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b\": container with ID starting with 3f004f1a26cc765b870a8856f3e1d97ac53712c0a7392b083868d7911930c86b not found: ID does not exist"
Oct 30 19:33:10.055112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.055085    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvvj\" (UniqueName: \"kubernetes.io/projected/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-kube-api-access-frvvj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:10.055112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.055111    5085 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc-extensions-socket-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:10.144147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.144110    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:33:10.149677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.149652    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-vjj8v"]
Oct 30 19:33:10.630121 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:10.630091    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" path="/var/lib/kubelet/pods/1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc/volumes"
Oct 30 19:33:12.546142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546109    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"]
Oct 30 19:33:12.546529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546459    5085 cpu_manager.go:414] "containerMap: removing container" podUID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" containerName="authorino"
Oct 30 19:33:12.546529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546469    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" containerName="authorino"
Oct 30 19:33:12.546529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546498    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" containerName="manager"
Oct 30 19:33:12.546529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546504    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" containerName="manager"
Oct 30 19:33:12.546675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546594    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b72ce3-a4b3-4920-bab8-e3ce8b65322a" containerName="authorino"
Oct 30 19:33:12.546675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.546607    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a006a13-cb81-4f70-b3c7-53f7dc7a5dbc" containerName="manager"
Oct 30 19:33:12.581786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.581760    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.604166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.604137    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"]
Oct 30 19:33:12.631543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.631504    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:33:12.648620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.648592    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 19:33:12.648620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.648599    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:33:12.677185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.677161    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvpm\" (UniqueName: \"kubernetes.io/projected/976d229d-31d9-40b9-907b-0b459a957b82-kube-api-access-ffvpm\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.677273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.677192    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/976d229d-31d9-40b9-907b-0b459a957b82-data\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.677801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.677786    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-g5hn4\""
Oct 30 19:33:12.777887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.777853    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvpm\" (UniqueName: \"kubernetes.io/projected/976d229d-31d9-40b9-907b-0b459a957b82-kube-api-access-ffvpm\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.777887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.777889    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/976d229d-31d9-40b9-907b-0b459a957b82-data\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.778208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.778193    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/976d229d-31d9-40b9-907b-0b459a957b82-data\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.836488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.836416    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvpm\" (UniqueName: \"kubernetes.io/projected/976d229d-31d9-40b9-907b-0b459a957b82-kube-api-access-ffvpm\") pod \"postgres-7cdbf6dc44-gldql\" (UID: \"976d229d-31d9-40b9-907b-0b459a957b82\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:12.908623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:12.908591    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:13.293525 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:13.293490    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"]
Oct 30 19:33:13.296911 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:33:13.296875    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976d229d_31d9_40b9_907b_0b459a957b82.slice/crio-bf33240f0bc23942df5cba6f6aa87b9c55aa9affba28b8b13d34c24357671e67 WatchSource:0}: Error finding container bf33240f0bc23942df5cba6f6aa87b9c55aa9affba28b8b13d34c24357671e67: Status 404 returned error can't find the container with id bf33240f0bc23942df5cba6f6aa87b9c55aa9affba28b8b13d34c24357671e67
Oct 30 19:33:14.029904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:14.029868    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql" event={"ID":"976d229d-31d9-40b9-907b-0b459a957b82","Type":"ContainerStarted","Data":"bf33240f0bc23942df5cba6f6aa87b9c55aa9affba28b8b13d34c24357671e67"}
Oct 30 19:33:21.945307 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:21.945262    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:33:26.094544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:26.094485    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql" event={"ID":"976d229d-31d9-40b9-907b-0b459a957b82","Type":"ContainerStarted","Data":"967dc5783fb90e47b892099ce545a59e36008b6e21975559bd3714d87da96386"}
Oct 30 19:33:26.095026 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:26.094592    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:26.206441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:26.206387    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql" podStartSLOduration=1.544831144 podStartE2EDuration="14.206373146s" totalImagesPullingTime="12.661542002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:26.20452634 +0000 UTC m=+515.708930068" watchObservedRunningTime="2026-10-30 19:33:26.206373146 +0000 UTC m=+515.710776897"
Oct 30 19:33:32.129249 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:32.129213    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-gldql"
Oct 30 19:33:39.242407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.242370    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:33:39.246387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.246366    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.296318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.296277    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:33:39.296483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.296364    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:33:39.314716 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.314688    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.314716 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.314717    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzmr\" (UniqueName: \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.314855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.314832    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.342937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.342903    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:33:39.342937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.342912    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:33:39.343128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.343048    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:33:39.343128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.343120    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jg4kd\""
Oct 30 19:33:39.345543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.345525    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:33:39.415498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.415466    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.415799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.415644    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzmr\" (UniqueName: \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.415799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.415743    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.415799 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:39.415764    5085 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Oct 30 19:33:39.415977 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:39.415856    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert podName:7d9dbfd8-8b14-4c55-9adb-8764a5d281e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:33:39.915832348 +0000 UTC m=+529.420236073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert") pod "maas-controller-6fc6f45d55-zjfj5" (UID: "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5") : secret "maas-controller-webhook-cert" not found
Oct 30 19:33:39.418320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.418293    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.474375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.474344    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzmr\" (UniqueName: \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.920112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.920073    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:39.922428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:39.922408    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") pod \"maas-controller-6fc6f45d55-zjfj5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") " pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:40.161246 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:40.161216    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:40.574211 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:40.574189    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:33:40.576259 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:33:40.576227    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9dbfd8_8b14_4c55_9adb_8764a5d281e5.slice/crio-0b43299c4ebe190d883dca17324a85ac6b059aede1e11b49d01eb521003edadc WatchSource:0}: Error finding container 0b43299c4ebe190d883dca17324a85ac6b059aede1e11b49d01eb521003edadc: Status 404 returned error can't find the container with id 0b43299c4ebe190d883dca17324a85ac6b059aede1e11b49d01eb521003edadc
Oct 30 19:33:41.169354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:41.169316    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" event={"ID":"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5","Type":"ContainerStarted","Data":"0b43299c4ebe190d883dca17324a85ac6b059aede1e11b49d01eb521003edadc"}
Oct 30 19:33:47.211252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:47.211214    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" event={"ID":"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5","Type":"ContainerStarted","Data":"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"}
Oct 30 19:33:47.211744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:47.211291    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:47.327002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:47.326945    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" podStartSLOduration=2.107605464 podStartE2EDuration="8.326929941s" totalImagesPullingTime="6.219324477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:33:47.324127605 +0000 UTC m=+536.828531352" watchObservedRunningTime="2026-10-30 19:33:47.326929941 +0000 UTC m=+536.831333685"
Oct 30 19:33:51.937666 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:51.937629    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:33:58.218133 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.218099    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:58.599937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.599857    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:33:58.600102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.600081    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" podUID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" containerName="manager" containerID="cri-o://6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6" gracePeriod=30
Oct 30 19:33:58.853733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.853670    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:58.892579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.892526    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") pod \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") "
Oct 30 19:33:58.892736 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.892600    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\" (UniqueName: \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\") pod \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") "
Oct 30 19:33:58.892736 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.892627    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\") pod \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\" (UID: \"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5\") "
Oct 30 19:33:58.894860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.894832    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert" pod "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" (UID: "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:58.894981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.894854    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs" pod "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" (UID: "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:58.894981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.894870    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr" pod "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" (UID: "7d9dbfd8-8b14-4c55-9adb-8764a5d281e5"). InnerVolumeSpecName "kube-api-access-9nzmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:58.993488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.993456    5085 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-webhook-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:58.993488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.993484    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzmr\" (UniqueName: \"kubernetes.io/projected/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-kube-api-access-9nzmr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:58.993488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:58.993495    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:59.281807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.281772    5085 generic.go:356] "Generic (PLEG): container finished" podID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" containerID="6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6" exitCode=0
Oct 30 19:33:59.282284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.281826    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5"
Oct 30 19:33:59.282284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.281853    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" event={"ID":"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5","Type":"ContainerDied","Data":"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"}
Oct 30 19:33:59.282284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.281907    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fc6f45d55-zjfj5" event={"ID":"7d9dbfd8-8b14-4c55-9adb-8764a5d281e5","Type":"ContainerDied","Data":"0b43299c4ebe190d883dca17324a85ac6b059aede1e11b49d01eb521003edadc"}
Oct 30 19:33:59.282284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.281933    5085 scope.go:122] "RemoveContainer" containerID="6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"
Oct 30 19:33:59.291893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.291876    5085 scope.go:122] "RemoveContainer" containerID="6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"
Oct 30 19:33:59.292182 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:33:59.292166    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6\": container with ID starting with 6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6 not found: ID does not exist" containerID="6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"
Oct 30 19:33:59.292264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.292190    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6"} err="failed to get container status \"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6\": rpc error: code = NotFound desc = could not find container \"6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6\": container with ID starting with 6bcf83d65e2a3ddbce19c9f157e931cda1de72e643b49038f1b484b888b3e7b6 not found: ID does not exist"
Oct 30 19:33:59.437591 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.437541    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:33:59.483981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:33:59.483950    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fc6f45d55-zjfj5"]
Oct 30 19:34:00.361212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.361179    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xwf7b"]
Oct 30 19:34:00.361615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.361593    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" containerName="manager"
Oct 30 19:34:00.361615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.361605    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" containerName="manager"
Oct 30 19:34:00.361698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.361662    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" containerName="manager"
Oct 30 19:34:00.363794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.363775    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.407056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.407022    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn47x\" (UniqueName: \"kubernetes.io/projected/f7e56e43-f4e7-4719-9c01-2e8d228a773f-kube-api-access-sn47x\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.407195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.407177    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7e56e43-f4e7-4719-9c01-2e8d228a773f-cert\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.413974 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.413948    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:34:00.414084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.414059    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vmjs7\""
Oct 30 19:34:00.414131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.414117    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:34:00.414178 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.414137    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 19:34:00.420269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.420246    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xwf7b"]
Oct 30 19:34:00.508236 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.508208    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7e56e43-f4e7-4719-9c01-2e8d228a773f-cert\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.508402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.508246    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn47x\" (UniqueName: \"kubernetes.io/projected/f7e56e43-f4e7-4719-9c01-2e8d228a773f-kube-api-access-sn47x\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.510765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.510737    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7e56e43-f4e7-4719-9c01-2e8d228a773f-cert\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.568119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.568092    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn47x\" (UniqueName: \"kubernetes.io/projected/f7e56e43-f4e7-4719-9c01-2e8d228a773f-kube-api-access-sn47x\") pod \"kserve-controller-manager-776cc66d47-xwf7b\" (UID: \"f7e56e43-f4e7-4719-9c01-2e8d228a773f\") " pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.570605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.570578    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x4rdp"]
Oct 30 19:34:00.573443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.573423    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.609319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.609289    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b387ef-f51d-416a-a75e-ce2560713559-tls-certs\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.609424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.609326    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpl8m\" (UniqueName: \"kubernetes.io/projected/f7b387ef-f51d-416a-a75e-ce2560713559-kube-api-access-gpl8m\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.630399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.630340    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9dbfd8-8b14-4c55-9adb-8764a5d281e5" path="/var/lib/kubelet/pods/7d9dbfd8-8b14-4c55-9adb-8764a5d281e5/volumes"
Oct 30 19:34:00.630646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.630631    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x4rdp"]
Oct 30 19:34:00.631722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.631702    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-47gjz\""
Oct 30 19:34:00.631893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.631874    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 19:34:00.675149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.675120    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:00.695202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.695174    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-pc2dt"]
Oct 30 19:34:00.699270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.699250    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.709889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.709867    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b387ef-f51d-416a-a75e-ce2560713559-tls-certs\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.710000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.709913    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpl8m\" (UniqueName: \"kubernetes.io/projected/f7b387ef-f51d-416a-a75e-ce2560713559-kube-api-access-gpl8m\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.712459 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.712433    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7b387ef-f51d-416a-a75e-ce2560713559-tls-certs\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.750670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.750634    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 19:34:00.750821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.750691    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-vpgnb\""
Oct 30 19:34:00.750821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.750732    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 19:34:00.761679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.761646    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pc2dt"]
Oct 30 19:34:00.769640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.769614    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpl8m\" (UniqueName: \"kubernetes.io/projected/f7b387ef-f51d-416a-a75e-ce2560713559-kube-api-access-gpl8m\") pod \"model-serving-api-75bd89fdfc-x4rdp\" (UID: \"f7b387ef-f51d-416a-a75e-ce2560713559\") " pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.810842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.810814    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-metrics-certs\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.810980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.810845    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtt8\" (UniqueName: \"kubernetes.io/projected/e2f10a7d-5fae-4943-9d9e-c13705842b79-kube-api-access-xdtt8\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.810980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.810880    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-cert\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.887358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.887293    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:00.911755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.911728    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-cert\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.911874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.911828    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-metrics-certs\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.911874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.911850    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtt8\" (UniqueName: \"kubernetes.io/projected/e2f10a7d-5fae-4943-9d9e-c13705842b79-kube-api-access-xdtt8\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.914119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.914096    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-cert\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.914324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.914302    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2f10a7d-5fae-4943-9d9e-c13705842b79-metrics-certs\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:00.970428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:00.970397    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtt8\" (UniqueName: \"kubernetes.io/projected/e2f10a7d-5fae-4943-9d9e-c13705842b79-kube-api-access-xdtt8\") pod \"odh-model-controller-769875db56-pc2dt\" (UID: \"e2f10a7d-5fae-4943-9d9e-c13705842b79\") " pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:01.012189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.012160    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:01.046688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.046663    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-xwf7b"]
Oct 30 19:34:01.048755 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:34:01.048726    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e56e43_f4e7_4719_9c01_2e8d228a773f.slice/crio-7c9c4977a62b691e89224550329ec0380b3ec60d18cdbbdd8c6ef7bbac779eeb WatchSource:0}: Error finding container 7c9c4977a62b691e89224550329ec0380b3ec60d18cdbbdd8c6ef7bbac779eeb: Status 404 returned error can't find the container with id 7c9c4977a62b691e89224550329ec0380b3ec60d18cdbbdd8c6ef7bbac779eeb
Oct 30 19:34:01.258370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.258347    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-x4rdp"]
Oct 30 19:34:01.260343 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:34:01.260315    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b387ef_f51d_416a_a75e_ce2560713559.slice/crio-cc259b8a6a83cec879e610cc84dda160606e872e24252b54ba3adfbd88f50b35 WatchSource:0}: Error finding container cc259b8a6a83cec879e610cc84dda160606e872e24252b54ba3adfbd88f50b35: Status 404 returned error can't find the container with id cc259b8a6a83cec879e610cc84dda160606e872e24252b54ba3adfbd88f50b35
Oct 30 19:34:01.294361 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.294327    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b" event={"ID":"f7e56e43-f4e7-4719-9c01-2e8d228a773f","Type":"ContainerStarted","Data":"7c9c4977a62b691e89224550329ec0380b3ec60d18cdbbdd8c6ef7bbac779eeb"}
Oct 30 19:34:01.295591 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.295543    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp" event={"ID":"f7b387ef-f51d-416a-a75e-ce2560713559","Type":"ContainerStarted","Data":"cc259b8a6a83cec879e610cc84dda160606e872e24252b54ba3adfbd88f50b35"}
Oct 30 19:34:01.379149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:01.379115    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pc2dt"]
Oct 30 19:34:01.382943 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:34:01.382902    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f10a7d_5fae_4943_9d9e_c13705842b79.slice/crio-fc1b16ea99b857836ec582a3f27ee897df7939e6c09a25c3c37482d1636e5293 WatchSource:0}: Error finding container fc1b16ea99b857836ec582a3f27ee897df7939e6c09a25c3c37482d1636e5293: Status 404 returned error can't find the container with id fc1b16ea99b857836ec582a3f27ee897df7939e6c09a25c3c37482d1636e5293
Oct 30 19:34:02.305301 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:02.305261    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pc2dt" event={"ID":"e2f10a7d-5fae-4943-9d9e-c13705842b79","Type":"ContainerStarted","Data":"fc1b16ea99b857836ec582a3f27ee897df7939e6c09a25c3c37482d1636e5293"}
Oct 30 19:34:04.321182 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:04.321108    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp" event={"ID":"f7b387ef-f51d-416a-a75e-ce2560713559","Type":"ContainerStarted","Data":"0d44fd102940795370e3c0b9f9b079e1902a5deebc0092cfe1c0fbbb720004fe"}
Oct 30 19:34:04.321703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:04.321264    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:04.435406 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:04.435343    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp" podStartSLOduration=1.5651242760000001 podStartE2EDuration="4.435324412s" totalImagesPullingTime="2.870200136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:04.432603801 +0000 UTC m=+553.937007545" watchObservedRunningTime="2026-10-30 19:34:04.435324412 +0000 UTC m=+553.939728163"
Oct 30 19:34:05.327523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:05.327422    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b" event={"ID":"f7e56e43-f4e7-4719-9c01-2e8d228a773f","Type":"ContainerStarted","Data":"72608d4f32b92ad905983a9211933528480c7cce506f82191998e6e19d1ef65c"}
Oct 30 19:34:05.327523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:05.327485    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:05.329095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:05.329063    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pc2dt" event={"ID":"e2f10a7d-5fae-4943-9d9e-c13705842b79","Type":"ContainerStarted","Data":"508fb02416522bc3aab764d7a5538f258851e93a29b73250e9ce9fe442231379"}
Oct 30 19:34:05.438836 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:05.438785    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b" podStartSLOduration=1.485858541 podStartE2EDuration="5.438771533s" totalImagesPullingTime="3.952912992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:05.437103694 +0000 UTC m=+554.941507441" watchObservedRunningTime="2026-10-30 19:34:05.438771533 +0000 UTC m=+554.943175280"
Oct 30 19:34:05.548312 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:05.548258    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-pc2dt" podStartSLOduration=1.772656362 podStartE2EDuration="5.548240854s" totalImagesPullingTime="3.775584492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:05.547465435 +0000 UTC m=+555.051869183" watchObservedRunningTime="2026-10-30 19:34:05.548240854 +0000 UTC m=+555.052644604"
Oct 30 19:34:06.334037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:06.334007    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:15.333746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:15.333712    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-x4rdp"
Oct 30 19:34:17.341546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:17.341510    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-pc2dt"
Oct 30 19:34:21.938703 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:34:21.938608    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:34:29.559331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:29.559247    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:34:36.339549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:36.339516    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-xwf7b"
Oct 30 19:34:38.560966 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.560932    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:38.564354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.564335    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:38.619358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.619328    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:38.665493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.665459    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s748\" (UniqueName: \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\") pod \"authorino-6f6944bcdd-wvh89\" (UID: \"86ca10f4-628d-4f27-9cb0-191a8a270e6a\") " pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:38.766165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.766082    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s748\" (UniqueName: \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\") pod \"authorino-6f6944bcdd-wvh89\" (UID: \"86ca10f4-628d-4f27-9cb0-191a8a270e6a\") " pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:38.824754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.824682    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s748\" (UniqueName: \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\") pod \"authorino-6f6944bcdd-wvh89\" (UID: \"86ca10f4-628d-4f27-9cb0-191a8a270e6a\") " pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:38.877909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.877876    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:38.943063 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.943031    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:38.976922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.976894    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:38.979913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:38.979894    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:39.034579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.034536    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:39.068916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.068883    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9f7\" (UniqueName: \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\") pod \"authorino-d955c5b5f-7bctz\" (UID: \"2a7897ae-e627-4c87-98e9-6775a4e06e4c\") " pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:39.169862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.169832    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9f7\" (UniqueName: \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\") pod \"authorino-d955c5b5f-7bctz\" (UID: \"2a7897ae-e627-4c87-98e9-6775a4e06e4c\") " pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:39.228613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.228575    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9f7\" (UniqueName: \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\") pod \"authorino-d955c5b5f-7bctz\" (UID: \"2a7897ae-e627-4c87-98e9-6775a4e06e4c\") " pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:39.244549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.244528    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:39.247085 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:34:39.247059    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ca10f4_628d_4f27_9cb0_191a8a270e6a.slice/crio-48d69754499cf41781bdeab3e74c47d08c736fd52479ba724b2faf9313370a1c WatchSource:0}: Error finding container 48d69754499cf41781bdeab3e74c47d08c736fd52479ba724b2faf9313370a1c: Status 404 returned error can't find the container with id 48d69754499cf41781bdeab3e74c47d08c736fd52479ba724b2faf9313370a1c
Oct 30 19:34:39.291620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.291590    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:39.371650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.371617    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:39.406989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.406959    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:34:39.410116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.410097    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.463610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.463511    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 19:34:39.464916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.464890    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:34:39.473002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.472979    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.473104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.473063    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxgk\" (UniqueName: \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.529857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.529826    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" event={"ID":"86ca10f4-628d-4f27-9cb0-191a8a270e6a","Type":"ContainerStarted","Data":"48d69754499cf41781bdeab3e74c47d08c736fd52479ba724b2faf9313370a1c"}
Oct 30 19:34:39.531352 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.531321    5085 generic.go:356] "Generic (PLEG): container finished" podID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerID="2de2d7f4258015d7465c595f86f0fd8f647852485ca89bf7fb1a43d652f49887" exitCode=0
Oct 30 19:34:39.531493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.531396    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" event={"ID":"f7eff08b-642e-4125-90e1-88ea8adf5385","Type":"ContainerDied","Data":"2de2d7f4258015d7465c595f86f0fd8f647852485ca89bf7fb1a43d652f49887"}
Oct 30 19:34:39.531826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.531809    5085 scope.go:122] "RemoveContainer" containerID="2de2d7f4258015d7465c595f86f0fd8f647852485ca89bf7fb1a43d652f49887"
Oct 30 19:34:39.533997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.533960    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:34:39.534131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.534111    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:34:39.534227 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.534210    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:34:39.574128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.574096    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.574443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.574139    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxgk\" (UniqueName: \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.576780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.576756    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.632598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.632553    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxgk\" (UniqueName: \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\") pod \"authorino-7c655d95d5-bjwvw\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") " pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:39.679001 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.678965    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:39.721400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:39.721322    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:34:40.101879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.101450    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:34:40.104380 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:34:40.104353    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode32d2372_c610_4eb4_b7cd_348fd9388a07.slice/crio-c6769a4b630c515b1424c0dbba992f1ffd985e0768198826f64c8e07ca06a8fd WatchSource:0}: Error finding container c6769a4b630c515b1424c0dbba992f1ffd985e0768198826f64c8e07ca06a8fd: Status 404 returned error can't find the container with id c6769a4b630c515b1424c0dbba992f1ffd985e0768198826f64c8e07ca06a8fd
Oct 30 19:34:40.385962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.385866    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:34:40.537933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.537895    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" event={"ID":"e32d2372-c610-4eb4-b7cd-348fd9388a07","Type":"ContainerStarted","Data":"c6769a4b630c515b1424c0dbba992f1ffd985e0768198826f64c8e07ca06a8fd"}
Oct 30 19:34:40.539862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.539835    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" event={"ID":"f7eff08b-642e-4125-90e1-88ea8adf5385","Type":"ContainerStarted","Data":"fdea639e3e0b78984c29d4279c989655e2d0efc4507df3b8578f3cb9e3d3bcb1"}
Oct 30 19:34:40.541034 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.541016    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d955c5b5f-7bctz" event={"ID":"2a7897ae-e627-4c87-98e9-6775a4e06e4c","Type":"ContainerStarted","Data":"dcb052f3ec69be3933ad0281ce4ae41e82c7ddda567a76bf282b9c3a5beb01e2"}
Oct 30 19:34:40.834246 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.834208    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:34:40.835196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.835170    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:34:40.835303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:40.835215    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 19:34:41.549890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.549852    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d955c5b5f-7bctz" event={"ID":"2a7897ae-e627-4c87-98e9-6775a4e06e4c","Type":"ContainerStarted","Data":"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"}
Oct 30 19:34:41.550090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.550030    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d955c5b5f-7bctz" podUID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" containerName="authorino" containerID="cri-o://db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930" gracePeriod=30
Oct 30 19:34:41.552694 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.552663    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" event={"ID":"86ca10f4-628d-4f27-9cb0-191a8a270e6a","Type":"ContainerStarted","Data":"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"}
Oct 30 19:34:41.555520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.555493    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" event={"ID":"e32d2372-c610-4eb4-b7cd-348fd9388a07","Type":"ContainerStarted","Data":"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"}
Oct 30 19:34:41.555740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.555673    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" podUID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" containerName="authorino" containerID="cri-o://7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33" gracePeriod=30
Oct 30 19:34:41.683585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.682922    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d955c5b5f-7bctz" podStartSLOduration=2.216493638 podStartE2EDuration="3.682901481s" totalImagesPullingTime="1.466407843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:41.681777914 +0000 UTC m=+591.186181662" watchObservedRunningTime="2026-10-30 19:34:41.682901481 +0000 UTC m=+591.187305231"
Oct 30 19:34:41.794599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.793351    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" podStartSLOduration=1.481006496 podStartE2EDuration="2.793336993s" totalImagesPullingTime="1.312330497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:41.792457731 +0000 UTC m=+591.296861477" watchObservedRunningTime="2026-10-30 19:34:41.793336993 +0000 UTC m=+591.297740741"
Oct 30 19:34:41.826033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.825998    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:34:41.826265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.826223    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" podUID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" containerName="authorino" containerID="cri-o://0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e" gracePeriod=30
Oct 30 19:34:41.834833 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.834796    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:34:41.835268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.834856    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 19:34:41.907676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.907362    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" podStartSLOduration=2.149015623 podStartE2EDuration="3.907346649s" totalImagesPullingTime="1.758331026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:41.906036918 +0000 UTC m=+591.410440689" watchObservedRunningTime="2026-10-30 19:34:41.907346649 +0000 UTC m=+591.411750393"
Oct 30 19:34:41.940765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.940739    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:41.943775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.943752    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:41.999295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.999263    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\" (UniqueName: \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\") pod \"86ca10f4-628d-4f27-9cb0-191a8a270e6a\" (UID: \"86ca10f4-628d-4f27-9cb0-191a8a270e6a\") "
Oct 30 19:34:41.999489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:41.999318    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\" (UniqueName: \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\") pod \"2a7897ae-e627-4c87-98e9-6775a4e06e4c\" (UID: \"2a7897ae-e627-4c87-98e9-6775a4e06e4c\") "
Oct 30 19:34:42.001331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.001304    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748" pod "86ca10f4-628d-4f27-9cb0-191a8a270e6a" (UID: "86ca10f4-628d-4f27-9cb0-191a8a270e6a"). InnerVolumeSpecName "kube-api-access-7s748". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:42.001579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.001541    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7" pod "2a7897ae-e627-4c87-98e9-6775a4e06e4c" (UID: "2a7897ae-e627-4c87-98e9-6775a4e06e4c"). InnerVolumeSpecName "kube-api-access-qp9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:42.099600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.097798    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:34:42.100079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.100059    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s748\" (UniqueName: \"kubernetes.io/projected/86ca10f4-628d-4f27-9cb0-191a8a270e6a-kube-api-access-7s748\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:42.100132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.100082    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9f7\" (UniqueName: \"kubernetes.io/projected/2a7897ae-e627-4c87-98e9-6775a4e06e4c-kube-api-access-qp9f7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:42.201290 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.201210    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\" (UniqueName: \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\") pod \"d60224c2-498c-4ce3-a3df-1ecce93b0ed6\" (UID: \"d60224c2-498c-4ce3-a3df-1ecce93b0ed6\") "
Oct 30 19:34:42.203344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.203310    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8" pod "d60224c2-498c-4ce3-a3df-1ecce93b0ed6" (UID: "d60224c2-498c-4ce3-a3df-1ecce93b0ed6"). InnerVolumeSpecName "kube-api-access-mspq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:42.302651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.302613    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mspq8\" (UniqueName: \"kubernetes.io/projected/d60224c2-498c-4ce3-a3df-1ecce93b0ed6-kube-api-access-mspq8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:42.584641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.583488    5085 generic.go:356] "Generic (PLEG): container finished" podID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" containerID="7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33" exitCode=0
Oct 30 19:34:42.584641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.583593    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wvh89"
Oct 30 19:34:42.584641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.583664    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" event={"ID":"86ca10f4-628d-4f27-9cb0-191a8a270e6a","Type":"ContainerDied","Data":"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"}
Oct 30 19:34:42.584641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.583702    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wvh89" event={"ID":"86ca10f4-628d-4f27-9cb0-191a8a270e6a","Type":"ContainerDied","Data":"48d69754499cf41781bdeab3e74c47d08c736fd52479ba724b2faf9313370a1c"}
Oct 30 19:34:42.584641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.583736    5085 scope.go:122] "RemoveContainer" containerID="7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"
Oct 30 19:34:42.587904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.585679    5085 generic.go:356] "Generic (PLEG): container finished" podID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" containerID="0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e" exitCode=0
Oct 30 19:34:42.587904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.585778    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" event={"ID":"d60224c2-498c-4ce3-a3df-1ecce93b0ed6","Type":"ContainerDied","Data":"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"}
Oct 30 19:34:42.587904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.585811    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg" event={"ID":"d60224c2-498c-4ce3-a3df-1ecce93b0ed6","Type":"ContainerDied","Data":"0b3cc54d2a880a9cd5be5277019e2280dc3babbb3eabfaea5e4d42c2b43a68a6"}
Oct 30 19:34:42.587904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.585898    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-fstjg"
Oct 30 19:34:42.592219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.592200    5085 generic.go:356] "Generic (PLEG): container finished" podID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" containerID="db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930" exitCode=0
Oct 30 19:34:42.592326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.592303    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d955c5b5f-7bctz" event={"ID":"2a7897ae-e627-4c87-98e9-6775a4e06e4c","Type":"ContainerDied","Data":"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"}
Oct 30 19:34:42.592507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.592397    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d955c5b5f-7bctz" event={"ID":"2a7897ae-e627-4c87-98e9-6775a4e06e4c","Type":"ContainerDied","Data":"dcb052f3ec69be3933ad0281ce4ae41e82c7ddda567a76bf282b9c3a5beb01e2"}
Oct 30 19:34:42.592507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.592270    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d955c5b5f-7bctz"
Oct 30 19:34:42.604273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.604193    5085 scope.go:122] "RemoveContainer" containerID="7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"
Oct 30 19:34:42.604516 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:34:42.604496    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33\": container with ID starting with 7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33 not found: ID does not exist" containerID="7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"
Oct 30 19:34:42.604623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.604529    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33"} err="failed to get container status \"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33\": rpc error: code = NotFound desc = could not find container \"7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33\": container with ID starting with 7700ee4d8e97e77b981bb4b9c111e6a175f9a3a8cf8b123557398d54b1e02b33 not found: ID does not exist"
Oct 30 19:34:42.604623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.604552    5085 scope.go:122] "RemoveContainer" containerID="0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"
Oct 30 19:34:42.613325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.613308    5085 scope.go:122] "RemoveContainer" containerID="0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"
Oct 30 19:34:42.613550 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:34:42.613532    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e\": container with ID starting with 0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e not found: ID does not exist" containerID="0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"
Oct 30 19:34:42.613648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.613609    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e"} err="failed to get container status \"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e\": rpc error: code = NotFound desc = could not find container \"0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e\": container with ID starting with 0d7dfa406d5ef2f2918774c973f732340438c1e9115c8b3660ad0bab1544c43e not found: ID does not exist"
Oct 30 19:34:42.613648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.613634    5085 scope.go:122] "RemoveContainer" containerID="db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"
Oct 30 19:34:42.622278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.622251    5085 scope.go:122] "RemoveContainer" containerID="db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"
Oct 30 19:34:42.622530 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:34:42.622501    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930\": container with ID starting with db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930 not found: ID does not exist" containerID="db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"
Oct 30 19:34:42.622702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.622529    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930"} err="failed to get container status \"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930\": rpc error: code = NotFound desc = could not find container \"db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930\": container with ID starting with db73877224b9cfc7edd85f8dbe2124b5e93ba757e37674786204b72771d65930 not found: ID does not exist"
Oct 30 19:34:42.739802 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.739767    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:42.747293 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.747260    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wvh89"]
Oct 30 19:34:42.834962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.834864    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:34:42.834962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.834935    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 19:34:42.850658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.850627    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:34:42.855627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.855602    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-fstjg"]
Oct 30 19:34:42.977587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.973619    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:42.983811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:42.983781    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d955c5b5f-7bctz"]
Oct 30 19:34:43.834644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:43.834604    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:34:43.834810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:43.834655    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 19:34:44.629230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:44.629196    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" path="/var/lib/kubelet/pods/2a7897ae-e627-4c87-98e9-6775a4e06e4c/volumes"
Oct 30 19:34:44.629618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:44.629515    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" path="/var/lib/kubelet/pods/86ca10f4-628d-4f27-9cb0-191a8a270e6a/volumes"
Oct 30 19:34:44.629838 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:44.629825    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" path="/var/lib/kubelet/pods/d60224c2-498c-4ce3-a3df-1ecce93b0ed6/volumes"
Oct 30 19:34:44.834668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:44.834587    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:34:44.834668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:44.834642    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 30 19:34:45.837759 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:45.837730    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:34:45.838174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:45.838011    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:34:45.838676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:45.838659    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf"
Oct 30 19:34:50.677217 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:50.677189    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:34:50.680026 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:34:50.680003    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:34:51.938331 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:34:51.938292    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:35:21.938503 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:35:21.938469    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:35:51.938312 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:35:51.938272    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:35:58.188975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.188897    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:35:58.189353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189338    5085 cpu_manager.go:414] "containerMap: removing container" podUID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" containerName="authorino"
Oct 30 19:35:58.189399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189355    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" containerName="authorino"
Oct 30 19:35:58.189399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189373    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" containerName="authorino"
Oct 30 19:35:58.189399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189379    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" containerName="authorino"
Oct 30 19:35:58.189399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189390    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" containerName="authorino"
Oct 30 19:35:58.189399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189396    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" containerName="authorino"
Oct 30 19:35:58.189542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189458    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ca10f4-628d-4f27-9cb0-191a8a270e6a" containerName="authorino"
Oct 30 19:35:58.189542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189469    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7897ae-e627-4c87-98e9-6775a4e06e4c" containerName="authorino"
Oct 30 19:35:58.189542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.189476    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60224c2-498c-4ce3-a3df-1ecce93b0ed6" containerName="authorino"
Oct 30 19:35:58.191761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.191744    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.241690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.241662    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:35:58.251213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.251167    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlcq4\" (UniqueName: \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.251376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.251347    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.288099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.288065    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-j7525\""
Oct 30 19:35:58.288099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.288065    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 19:35:58.290281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.290253    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:35:58.352033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.351999    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.352190 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.352044    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlcq4\" (UniqueName: \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.354412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.354392    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.410674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.410641    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlcq4\" (UniqueName: \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\") pod \"ai-gateway-controller-7d85675f98-v9b7q\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.504419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.504340    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:35:58.870455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.870433    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:35:58.872733 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:35:58.872703    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b1d185_cc22_4493_b0e8_acd55bfabea5.slice/crio-6e5606b39a795dc8342daa78fa42f8b4ba78893cfd7830058a17f5969a2db337 WatchSource:0}: Error finding container 6e5606b39a795dc8342daa78fa42f8b4ba78893cfd7830058a17f5969a2db337: Status 404 returned error can't find the container with id 6e5606b39a795dc8342daa78fa42f8b4ba78893cfd7830058a17f5969a2db337
Oct 30 19:35:58.982598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:35:58.982546    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" event={"ID":"b1b1d185-cc22-4493-b0e8-acd55bfabea5","Type":"ContainerStarted","Data":"6e5606b39a795dc8342daa78fa42f8b4ba78893cfd7830058a17f5969a2db337"}
Oct 30 19:36:04.009690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:04.009651    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" event={"ID":"b1b1d185-cc22-4493-b0e8-acd55bfabea5","Type":"ContainerStarted","Data":"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"}
Oct 30 19:36:04.010169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:04.009779    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:36:04.122099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:04.122045    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" podStartSLOduration=1.7342588989999999 podStartE2EDuration="6.122024752s" totalImagesPullingTime="4.387765853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:04.120598846 +0000 UTC m=+673.625002590" watchObservedRunningTime="2026-10-30 19:36:04.122024752 +0000 UTC m=+673.626428499"
Oct 30 19:36:06.466686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.466649    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"]
Oct 30 19:36:06.469751 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.469727    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.519087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.519056    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 30 19:36:06.570740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.569786    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jtxfz\""
Oct 30 19:36:06.570740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.570676    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"]
Oct 30 19:36:06.614348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.614313    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-t5gqn"]
Oct 30 19:36:06.617078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.617056    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.623605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.623553    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec560d-02a4-4c34-80c4-4a7e58bd5874-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.623890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.623676    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw76\" (UniqueName: \"kubernetes.io/projected/23ec560d-02a4-4c34-80c4-4a7e58bd5874-kube-api-access-fnw76\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.659612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.659574    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-t5gqn"]
Oct 30 19:36:06.724885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.724798    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec560d-02a4-4c34-80c4-4a7e58bd5874-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.725161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.725134    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4dm\" (UniqueName: \"kubernetes.io/projected/049e0e86-319c-4386-99ee-f13821b9c079-kube-api-access-rs4dm\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.725308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.725288    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw76\" (UniqueName: \"kubernetes.io/projected/23ec560d-02a4-4c34-80c4-4a7e58bd5874-kube-api-access-fnw76\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.726324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.726282    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049e0e86-319c-4386-99ee-f13821b9c079-plugins-config-volume\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.726629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.726013    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec560d-02a4-4c34-80c4-4a7e58bd5874-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.785736 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.785705    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw76\" (UniqueName: \"kubernetes.io/projected/23ec560d-02a4-4c34-80c4-4a7e58bd5874-kube-api-access-fnw76\") pod \"payload-pre-processing-7b98bb678c-j5xnn\" (UID: \"23ec560d-02a4-4c34-80c4-4a7e58bd5874\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:06.827021 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.826985    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049e0e86-319c-4386-99ee-f13821b9c079-plugins-config-volume\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.827204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.827082    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4dm\" (UniqueName: \"kubernetes.io/projected/049e0e86-319c-4386-99ee-f13821b9c079-kube-api-access-rs4dm\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.827730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.827709    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/049e0e86-319c-4386-99ee-f13821b9c079-plugins-config-volume\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.885472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.885436    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4dm\" (UniqueName: \"kubernetes.io/projected/049e0e86-319c-4386-99ee-f13821b9c079-kube-api-access-rs4dm\") pod \"payload-processing-5f78687d5b-t5gqn\" (UID: \"049e0e86-319c-4386-99ee-f13821b9c079\") " pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:06.930388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:06.930353    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:07.082203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:07.082173    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:07.315221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:07.315198    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-t5gqn"]
Oct 30 19:36:07.317820 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:07.317790    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049e0e86_319c_4386_99ee_f13821b9c079.slice/crio-c036a0a92056b37606c6c697e65c2c0f58ebb4013f274d79e3ab34c13f59d699 WatchSource:0}: Error finding container c036a0a92056b37606c6c697e65c2c0f58ebb4013f274d79e3ab34c13f59d699: Status 404 returned error can't find the container with id c036a0a92056b37606c6c697e65c2c0f58ebb4013f274d79e3ab34c13f59d699
Oct 30 19:36:07.451382 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:07.451285    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"]
Oct 30 19:36:07.454098 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:07.454072    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ec560d_02a4_4c34_80c4_4a7e58bd5874.slice/crio-3e7e956c78a6a02037ff14d736b84ba15b995fccf1d809e496346a205bad4c63 WatchSource:0}: Error finding container 3e7e956c78a6a02037ff14d736b84ba15b995fccf1d809e496346a205bad4c63: Status 404 returned error can't find the container with id 3e7e956c78a6a02037ff14d736b84ba15b995fccf1d809e496346a205bad4c63
Oct 30 19:36:08.031639 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:08.031597    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn" event={"ID":"23ec560d-02a4-4c34-80c4-4a7e58bd5874","Type":"ContainerStarted","Data":"3e7e956c78a6a02037ff14d736b84ba15b995fccf1d809e496346a205bad4c63"}
Oct 30 19:36:08.032887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:08.032861    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn" event={"ID":"049e0e86-319c-4386-99ee-f13821b9c079","Type":"ContainerStarted","Data":"c036a0a92056b37606c6c697e65c2c0f58ebb4013f274d79e3ab34c13f59d699"}
Oct 30 19:36:11.053157 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:11.053120    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn" event={"ID":"049e0e86-319c-4386-99ee-f13821b9c079","Type":"ContainerStarted","Data":"47b4b7cbee9bb58a8f08c7423859727c596cabfdfb64a8d57ef9626bca09415b"}
Oct 30 19:36:11.053647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:11.053212    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:11.166029 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:11.165941    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn" podStartSLOduration=1.653608057 podStartE2EDuration="5.165926396s" totalImagesPullingTime="3.512318339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:11.16366332 +0000 UTC m=+680.668067067" watchObservedRunningTime="2026-10-30 19:36:11.165926396 +0000 UTC m=+680.670330144"
Oct 30 19:36:12.059983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:12.059945    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn" event={"ID":"23ec560d-02a4-4c34-80c4-4a7e58bd5874","Type":"ContainerStarted","Data":"4299ef1fb915ae4234e61d7bbe3db86377e245ef1d3d8d6bab328154a112c4b4"}
Oct 30 19:36:12.060357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:12.060215    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:12.172006 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:12.171954    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn" podStartSLOduration=2.420679029 podStartE2EDuration="6.171939692s" totalImagesPullingTime="3.751260663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:12.170190726 +0000 UTC m=+681.674594470" watchObservedRunningTime="2026-10-30 19:36:12.171939692 +0000 UTC m=+681.676343457"
Oct 30 19:36:15.020842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.020811    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:36:15.376496 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.376418    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-76fc486898-xkrnb"]
Oct 30 19:36:15.379592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.379572    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.434970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.434932    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76fc486898-xkrnb"]
Oct 30 19:36:15.504509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.504473    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2gw\" (UniqueName: \"kubernetes.io/projected/0a4684b7-2357-4383-ac16-c5a6b3be17ee-kube-api-access-fz2gw\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.504711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.504521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4684b7-2357-4383-ac16-c5a6b3be17ee-metrics-certs\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.605445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.605409    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2gw\" (UniqueName: \"kubernetes.io/projected/0a4684b7-2357-4383-ac16-c5a6b3be17ee-kube-api-access-fz2gw\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.605719 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.605463    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4684b7-2357-4383-ac16-c5a6b3be17ee-metrics-certs\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.607922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.607899    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4684b7-2357-4383-ac16-c5a6b3be17ee-metrics-certs\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.663650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.663579    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2gw\" (UniqueName: \"kubernetes.io/projected/0a4684b7-2357-4383-ac16-c5a6b3be17ee-kube-api-access-fz2gw\") pod \"ai-gateway-controller-76fc486898-xkrnb\" (UID: \"0a4684b7-2357-4383-ac16-c5a6b3be17ee\") " pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:15.691424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:15.691393    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:16.056879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:16.056855    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76fc486898-xkrnb"]
Oct 30 19:36:16.059166 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:16.059124    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4684b7_2357_4383_ac16_c5a6b3be17ee.slice/crio-cb8bf71b7554484a351bbd4e36eb165d70d9937b7ef18634a352ee1895202347 WatchSource:0}: Error finding container cb8bf71b7554484a351bbd4e36eb165d70d9937b7ef18634a352ee1895202347: Status 404 returned error can't find the container with id cb8bf71b7554484a351bbd4e36eb165d70d9937b7ef18634a352ee1895202347
Oct 30 19:36:16.080925 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:16.080893    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb" event={"ID":"0a4684b7-2357-4383-ac16-c5a6b3be17ee","Type":"ContainerStarted","Data":"cb8bf71b7554484a351bbd4e36eb165d70d9937b7ef18634a352ee1895202347"}
Oct 30 19:36:17.088333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:17.088296    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb" event={"ID":"0a4684b7-2357-4383-ac16-c5a6b3be17ee","Type":"ContainerStarted","Data":"41af8dca58859dc16c7cb7ac753dbead29ce79024bc6044b6c631d2024e2caa7"}
Oct 30 19:36:17.088842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:17.088352    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:17.200787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:17.200722    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb" podStartSLOduration=1.367148993 podStartE2EDuration="2.200703484s" totalImagesPullingTime="833.554491ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:17.199093561 +0000 UTC m=+686.703497308" watchObservedRunningTime="2026-10-30 19:36:17.200703484 +0000 UTC m=+686.705107640"
Oct 30 19:36:21.938392 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:36:21.938358    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:36:22.063123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:22.063090    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-t5gqn"
Oct 30 19:36:23.067817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:23.067789    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-j5xnn"
Oct 30 19:36:28.096515 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.096480    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-76fc486898-xkrnb"
Oct 30 19:36:28.240087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.240052    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:36:28.240293 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.240271    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" podUID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" containerName="manager" containerID="cri-o://e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17" gracePeriod=30
Oct 30 19:36:28.489667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.489642    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:36:28.621746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.621658    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\") pod \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") "
Oct 30 19:36:28.621746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.621733    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\" (UniqueName: \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\") pod \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\" (UID: \"b1b1d185-cc22-4493-b0e8-acd55bfabea5\") "
Oct 30 19:36:28.623870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.623841    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4" pod "b1b1d185-cc22-4493-b0e8-acd55bfabea5" (UID: "b1b1d185-cc22-4493-b0e8-acd55bfabea5"). InnerVolumeSpecName "kube-api-access-xlcq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:28.624018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.623878    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs" pod "b1b1d185-cc22-4493-b0e8-acd55bfabea5" (UID: "b1b1d185-cc22-4493-b0e8-acd55bfabea5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:28.723556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.723525    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b1d185-cc22-4493-b0e8-acd55bfabea5-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:28.723556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:28.723552    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlcq4\" (UniqueName: \"kubernetes.io/projected/b1b1d185-cc22-4493-b0e8-acd55bfabea5-kube-api-access-xlcq4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:29.151826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.151792    5085 generic.go:356] "Generic (PLEG): container finished" podID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" containerID="e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17" exitCode=0
Oct 30 19:36:29.152213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.151830    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q"
Oct 30 19:36:29.152213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.151877    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" event={"ID":"b1b1d185-cc22-4493-b0e8-acd55bfabea5","Type":"ContainerDied","Data":"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"}
Oct 30 19:36:29.152213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.151927    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-v9b7q" event={"ID":"b1b1d185-cc22-4493-b0e8-acd55bfabea5","Type":"ContainerDied","Data":"6e5606b39a795dc8342daa78fa42f8b4ba78893cfd7830058a17f5969a2db337"}
Oct 30 19:36:29.152213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.151950    5085 scope.go:122] "RemoveContainer" containerID="e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"
Oct 30 19:36:29.160821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.160789    5085 scope.go:122] "RemoveContainer" containerID="e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"
Oct 30 19:36:29.161108 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:36:29.161082    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17\": container with ID starting with e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17 not found: ID does not exist" containerID="e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"
Oct 30 19:36:29.161172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.161117    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17"} err="failed to get container status \"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17\": rpc error: code = NotFound desc = could not find container \"e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17\": container with ID starting with e645c521fc910e742cfc81ffff410d575ea6953f5620aa627e2abee7fd8b0a17 not found: ID does not exist"
Oct 30 19:36:29.310332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.310291    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:36:29.357166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:29.357136    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-v9b7q"]
Oct 30 19:36:30.630484 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:30.630453    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" path="/var/lib/kubelet/pods/b1b1d185-cc22-4493-b0e8-acd55bfabea5/volumes"
Oct 30 19:36:38.449811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.449772    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"]
Oct 30 19:36:38.450228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.450214    5085 cpu_manager.go:414] "containerMap: removing container" podUID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" containerName="manager"
Oct 30 19:36:38.450289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.450229    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" containerName="manager"
Oct 30 19:36:38.450324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.450292    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b1d185-cc22-4493-b0e8-acd55bfabea5" containerName="manager"
Oct 30 19:36:38.454635 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.454616    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.504514 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.504487    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:36:38.507033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.506996    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.507155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.507112    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5pw\" (UniqueName: \"kubernetes.io/projected/0f6cb9b2-60f1-4817-813c-24c9850d6075-kube-api-access-qs5pw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.507200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.507167    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6cb9b2-60f1-4817-813c-24c9850d6075-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.507235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.507206    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.507235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.507228    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.507308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.507296    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.516654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.516617    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:36:38.551059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.551028    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-85qgn\""
Oct 30 19:36:38.551059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.551054    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 19:36:38.553417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.553396    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"]
Oct 30 19:36:38.607954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.607922    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5pw\" (UniqueName: \"kubernetes.io/projected/0f6cb9b2-60f1-4817-813c-24c9850d6075-kube-api-access-qs5pw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.607969    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6cb9b2-60f1-4817-813c-24c9850d6075-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608003    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608021    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608042    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608089    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608447    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608500    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.608621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.608534    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.610310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.610289    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f6cb9b2-60f1-4817-813c-24c9850d6075-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.610573 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.610543    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f6cb9b2-60f1-4817-813c-24c9850d6075-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.666393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.666357    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5pw\" (UniqueName: \"kubernetes.io/projected/0f6cb9b2-60f1-4817-813c-24c9850d6075-kube-api-access-qs5pw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9\" (UID: \"0f6cb9b2-60f1-4817-813c-24c9850d6075\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:38.767419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:38.767336    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:39.139668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:39.139637    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"]
Oct 30 19:36:39.141656 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:39.141617    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6cb9b2_60f1_4817_813c_24c9850d6075.slice/crio-6ba66ca34958fe42eb1b26a9cfb5ba4d17669d20969f75b73bbe7ddb324ab687 WatchSource:0}: Error finding container 6ba66ca34958fe42eb1b26a9cfb5ba4d17669d20969f75b73bbe7ddb324ab687: Status 404 returned error can't find the container with id 6ba66ca34958fe42eb1b26a9cfb5ba4d17669d20969f75b73bbe7ddb324ab687
Oct 30 19:36:39.143071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:39.143052    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:36:39.207633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:39.207603    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9" event={"ID":"0f6cb9b2-60f1-4817-813c-24c9850d6075","Type":"ContainerStarted","Data":"6ba66ca34958fe42eb1b26a9cfb5ba4d17669d20969f75b73bbe7ddb324ab687"}
Oct 30 19:36:42.044656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:42.044622    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:36:42.229355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:42.229321    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9" event={"ID":"0f6cb9b2-60f1-4817-813c-24c9850d6075","Type":"ContainerStarted","Data":"09b371ab5305ac4d473f5a9aa07cc18dfd5023063b6e746667d5e29a590032f9"}
Oct 30 19:36:42.343114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:42.343003    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9" podStartSLOduration=2.082381307 podStartE2EDuration="4.342986538s" totalImagesPullingTime="2.260605231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:42.340995249 +0000 UTC m=+711.845398996" watchObservedRunningTime="2026-10-30 19:36:42.342986538 +0000 UTC m=+711.847390282"
Oct 30 19:36:42.767735 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:42.767698    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:46.224271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.224231    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"]
Oct 30 19:36:46.226928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.226904    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275263    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275311    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275437    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq4j\" (UniqueName: \"kubernetes.io/projected/638c345b-06f3-400f-8c71-f6bfe6dd98bf-kube-api-access-nnq4j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275483    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275588    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.275769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.275661    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.327618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.327581    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 30 19:36:46.329906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.329874    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"]
Oct 30 19:36:46.376688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376654    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.376688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376693    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.376910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376752    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.376910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376775    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.376910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376792    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.376910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.376853    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq4j\" (UniqueName: \"kubernetes.io/projected/638c345b-06f3-400f-8c71-f6bfe6dd98bf-kube-api-access-nnq4j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.377213 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.377190    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.377294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.377205    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.377294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.377260    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.378927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.378899    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638c345b-06f3-400f-8c71-f6bfe6dd98bf-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.379328 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.379310    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638c345b-06f3-400f-8c71-f6bfe6dd98bf-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.435512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.435472    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq4j\" (UniqueName: \"kubernetes.io/projected/638c345b-06f3-400f-8c71-f6bfe6dd98bf-kube-api-access-nnq4j\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-b5vhn\" (UID: \"638c345b-06f3-400f-8c71-f6bfe6dd98bf\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.539705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.539622    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:46.913137 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:46.913109    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"]
Oct 30 19:36:46.915212 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:46.915180    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638c345b_06f3_400f_8c71_f6bfe6dd98bf.slice/crio-c6177b754fd0a36e77d899792a9e878d19b8d1dae24433c14a81018723af2a79 WatchSource:0}: Error finding container c6177b754fd0a36e77d899792a9e878d19b8d1dae24433c14a81018723af2a79: Status 404 returned error can't find the container with id c6177b754fd0a36e77d899792a9e878d19b8d1dae24433c14a81018723af2a79
Oct 30 19:36:47.256610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:47.256543    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn" event={"ID":"638c345b-06f3-400f-8c71-f6bfe6dd98bf","Type":"ContainerStarted","Data":"c6177b754fd0a36e77d899792a9e878d19b8d1dae24433c14a81018723af2a79"}
Oct 30 19:36:48.268124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:48.268086    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn" event={"ID":"638c345b-06f3-400f-8c71-f6bfe6dd98bf","Type":"ContainerStarted","Data":"38c3b76fe881a6f5d35a69aac7293f712a946d779fbfcc94848f63837bd5a72a"}
Oct 30 19:36:48.380810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:48.380758    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn" podStartSLOduration=1.738157497 podStartE2EDuration="2.380743665s" totalImagesPullingTime="642.586168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:48.379520119 +0000 UTC m=+717.883923878" watchObservedRunningTime="2026-10-30 19:36:48.380743665 +0000 UTC m=+717.885147446"
Oct 30 19:36:48.540587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:48.540489    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:48.768021 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:48.767985    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:48.780119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:48.780093    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:49.284462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:49.284432    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9"
Oct 30 19:36:49.869314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:49.869276    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:36:51.938171 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:36:51.938124    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:36:52.157250 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.157216    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"]
Oct 30 19:36:52.160898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.160881    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229308    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d83157-4461-44cd-a4b3-804bc7089c77-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229367    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229482    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw5rm\" (UniqueName: \"kubernetes.io/projected/41d83157-4461-44cd-a4b3-804bc7089c77-kube-api-access-bw5rm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229621    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.229689 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.229671    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.257030 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.257002    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 19:36:52.259875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.259852    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"]
Oct 30 19:36:52.330799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330766    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d83157-4461-44cd-a4b3-804bc7089c77-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.330980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330822    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.330980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330846    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.330980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330863    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw5rm\" (UniqueName: \"kubernetes.io/projected/41d83157-4461-44cd-a4b3-804bc7089c77-kube-api-access-bw5rm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.330980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330888    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.330980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.330937    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.331366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.331344    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.331450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.331397    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.331511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.331441    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.333174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.333156    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41d83157-4461-44cd-a4b3-804bc7089c77-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.333302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.333285    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41d83157-4461-44cd-a4b3-804bc7089c77-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.389816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.389790    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw5rm\" (UniqueName: \"kubernetes.io/projected/41d83157-4461-44cd-a4b3-804bc7089c77-kube-api-access-bw5rm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qqqrm\" (UID: \"41d83157-4461-44cd-a4b3-804bc7089c77\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.472727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.472694    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:52.841906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:52.841871    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"]
Oct 30 19:36:52.843504 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:36:52.843474    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d83157_4461_44cd_a4b3_804bc7089c77.slice/crio-26087c94c1aa38beb8155504683edbd0138b9f4cf80b6f33654bd1c79b596e95 WatchSource:0}: Error finding container 26087c94c1aa38beb8155504683edbd0138b9f4cf80b6f33654bd1c79b596e95: Status 404 returned error can't find the container with id 26087c94c1aa38beb8155504683edbd0138b9f4cf80b6f33654bd1c79b596e95
Oct 30 19:36:53.302369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:53.302336    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm" event={"ID":"41d83157-4461-44cd-a4b3-804bc7089c77","Type":"ContainerStarted","Data":"26087c94c1aa38beb8155504683edbd0138b9f4cf80b6f33654bd1c79b596e95"}
Oct 30 19:36:54.308622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:54.308588    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm" event={"ID":"41d83157-4461-44cd-a4b3-804bc7089c77","Type":"ContainerStarted","Data":"f2da68afe13dad1a877cf463ca0d3746ebd98cea54e6f8898e526723f56f9974"}
Oct 30 19:36:54.422670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:54.422616    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm" podStartSLOduration=1.78437788 podStartE2EDuration="2.422602107s" totalImagesPullingTime="638.224227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:54.420299686 +0000 UTC m=+723.924703434" watchObservedRunningTime="2026-10-30 19:36:54.422602107 +0000 UTC m=+723.927005854"
Oct 30 19:36:54.473553 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:54.473523    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:36:56.543425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:56.542636    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:56.563126 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:56.562857    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:36:57.337990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:36:57.337952    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-b5vhn"
Oct 30 19:37:02.473532 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:02.473496    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:37:02.486027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:02.486002    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:37:03.368534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.368504    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qqqrm"
Oct 30 19:37:03.512841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.512805    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:37:03.723400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.723360    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"]
Oct 30 19:37:03.726723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.726699    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.779205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.779176    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 19:37:03.787730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.787699    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"]
Oct 30 19:37:03.838091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838050    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.838284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838116    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s229s\" (UniqueName: \"kubernetes.io/projected/17151fda-8d13-423b-9fba-b177eb796463-kube-api-access-s229s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.838284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838199    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.838409 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838286    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.838465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838434    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.838526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.838461    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17151fda-8d13-423b-9fba-b177eb796463-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939069    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939115    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939177    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939192    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17151fda-8d13-423b-9fba-b177eb796463-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939211    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939235    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s229s\" (UniqueName: \"kubernetes.io/projected/17151fda-8d13-423b-9fba-b177eb796463-kube-api-access-s229s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939576    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939647    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.939752 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.939716    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.941712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.941682    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17151fda-8d13-423b-9fba-b177eb796463-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.941867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.941849    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17151fda-8d13-423b-9fba-b177eb796463-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:03.997733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:03.997660    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s229s\" (UniqueName: \"kubernetes.io/projected/17151fda-8d13-423b-9fba-b177eb796463-kube-api-access-s229s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh\" (UID: \"17151fda-8d13-423b-9fba-b177eb796463\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:04.039661 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:04.039621    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:04.421932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:04.421899    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"]
Oct 30 19:37:04.426849 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:37:04.426815    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17151fda_8d13_423b_9fba_b177eb796463.slice/crio-dbf37f8849beb32e0fb748383f69f0af21f565fe3afdeb4cce172984114fd50b WatchSource:0}: Error finding container dbf37f8849beb32e0fb748383f69f0af21f565fe3afdeb4cce172984114fd50b: Status 404 returned error can't find the container with id dbf37f8849beb32e0fb748383f69f0af21f565fe3afdeb4cce172984114fd50b
Oct 30 19:37:05.373107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:05.373074    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh" event={"ID":"17151fda-8d13-423b-9fba-b177eb796463","Type":"ContainerStarted","Data":"c8fd541c20390c865037df92c7dddba1ad68cafb3061dd23933dd279aecd8391"}
Oct 30 19:37:05.373479 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:05.373137    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh" event={"ID":"17151fda-8d13-423b-9fba-b177eb796463","Type":"ContainerStarted","Data":"dbf37f8849beb32e0fb748383f69f0af21f565fe3afdeb4cce172984114fd50b"}
Oct 30 19:37:05.486023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:05.485975    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh" podStartSLOduration=1.667127551 podStartE2EDuration="2.485959813s" totalImagesPullingTime="818.832262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:05.484966438 +0000 UTC m=+734.989370184" watchObservedRunningTime="2026-10-30 19:37:05.485959813 +0000 UTC m=+734.990363560"
Oct 30 19:37:06.040044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:06.040015    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:07.070335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.070300    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"]
Oct 30 19:37:07.073113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.073089    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.122977 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.122929    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 19:37:07.173743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.173703    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"]
Oct 30 19:37:07.176599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176575    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.176706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176690    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.176740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176717    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.176799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176786    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a40e56-bd58-4960-98ab-e6f7be89e798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.176842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176813    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.176888 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.176871    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq7lv\" (UniqueName: \"kubernetes.io/projected/00a40e56-bd58-4960-98ab-e6f7be89e798-kube-api-access-sq7lv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278063    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a40e56-bd58-4960-98ab-e6f7be89e798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278115    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278135    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq7lv\" (UniqueName: \"kubernetes.io/projected/00a40e56-bd58-4960-98ab-e6f7be89e798-kube-api-access-sq7lv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278183    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278263    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278284    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278617    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278719    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.278823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.278805    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.280375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.280352    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00a40e56-bd58-4960-98ab-e6f7be89e798-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.280768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.280748    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a40e56-bd58-4960-98ab-e6f7be89e798-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.341965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.341895    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq7lv\" (UniqueName: \"kubernetes.io/projected/00a40e56-bd58-4960-98ab-e6f7be89e798-kube-api-access-sq7lv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th\" (UID: \"00a40e56-bd58-4960-98ab-e6f7be89e798\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.383745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.383717    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:07.754750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:07.754718    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"]
Oct 30 19:37:08.393574 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:08.390487    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" event={"ID":"00a40e56-bd58-4960-98ab-e6f7be89e798","Type":"ContainerStarted","Data":"1177319e653a722678504c4bdb7292ba855d38c65c87b3f3ae401a26f2e88092"}
Oct 30 19:37:09.397747 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:09.397713    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" event={"ID":"00a40e56-bd58-4960-98ab-e6f7be89e798","Type":"ContainerStarted","Data":"12d9b47dc30aa137461a34184a7e0fa6263bfe5d4101dc9075755f8911508ace"}
Oct 30 19:37:09.510849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:09.510787    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podStartSLOduration=1.8925986080000001 podStartE2EDuration="2.510771333s" totalImagesPullingTime="618.172725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:09.508891978 +0000 UTC m=+739.013295728" watchObservedRunningTime="2026-10-30 19:37:09.510771333 +0000 UTC m=+739.015175080"
Oct 30 19:37:10.384536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:10.384507    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:14.040387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:14.040344    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:14.052929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:14.052902    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:14.439317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:14.439288    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh"
Oct 30 19:37:17.384435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:17.384399    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:17.397128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:17.397096    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:17.455441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:17.455419    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:37:17.979698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:17.979662    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:37:21.938926 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:37:21.938874    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:37:33.010754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.010679    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fcd7f7c4-p4ccs"]
Oct 30 19:37:33.014308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.014288    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.064914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.064886    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcd7f7c4-p4ccs"]
Oct 30 19:37:33.128557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.128524    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-tls-cert\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.128799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.128657    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnvp\" (UniqueName: \"kubernetes.io/projected/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-kube-api-access-7lnvp\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.229338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.229299    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-tls-cert\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.229535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.229390    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnvp\" (UniqueName: \"kubernetes.io/projected/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-kube-api-access-7lnvp\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.231800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.231769    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-tls-cert\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.287950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.287881    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnvp\" (UniqueName: \"kubernetes.io/projected/559f8fac-bb0f-47c6-8bf7-12b98a1d50fb-kube-api-access-7lnvp\") pod \"authorino-5fcd7f7c4-p4ccs\" (UID: \"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb\") " pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.325526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.325491    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs"
Oct 30 19:37:33.695793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:33.695764    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcd7f7c4-p4ccs"]
Oct 30 19:37:33.697800 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:37:33.697764    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559f8fac_bb0f_47c6_8bf7_12b98a1d50fb.slice/crio-a06bc3d435244fc19b3b92c47de0649ecc11eed589f3798a3c69da553e5398c9 WatchSource:0}: Error finding container a06bc3d435244fc19b3b92c47de0649ecc11eed589f3798a3c69da553e5398c9: Status 404 returned error can't find the container with id a06bc3d435244fc19b3b92c47de0649ecc11eed589f3798a3c69da553e5398c9
Oct 30 19:37:34.538851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:34.538817    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs" event={"ID":"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb","Type":"ContainerStarted","Data":"a06bc3d435244fc19b3b92c47de0649ecc11eed589f3798a3c69da553e5398c9"}
Oct 30 19:37:36.552327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:36.552278    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs" event={"ID":"559f8fac-bb0f-47c6-8bf7-12b98a1d50fb","Type":"ContainerStarted","Data":"4251fcb4d92eb09e9de58a34ed49f50fd10777ee46fcaaee01b65b414451737a"}
Oct 30 19:37:36.661773 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:36.661723    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fcd7f7c4-p4ccs" podStartSLOduration=2.444691511 podStartE2EDuration="4.661706756s" totalImagesPullingTime="2.217015245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:36.660513522 +0000 UTC m=+766.164917269" watchObservedRunningTime="2026-10-30 19:37:36.661706756 +0000 UTC m=+766.166110503"
Oct 30 19:37:36.710872 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:36.710835    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:37:36.711088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:36.711060    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" podUID="e32d2372-c610-4eb4-b7cd-348fd9388a07" containerName="authorino" containerID="cri-o://dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2" gracePeriod=30
Oct 30 19:37:36.986173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:36.986142    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:37:37.069593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.069538    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\" (UniqueName: \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\") pod \"e32d2372-c610-4eb4-b7cd-348fd9388a07\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") "
Oct 30 19:37:37.069785 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.069745    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\" (UniqueName: \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\") pod \"e32d2372-c610-4eb4-b7cd-348fd9388a07\" (UID: \"e32d2372-c610-4eb4-b7cd-348fd9388a07\") "
Oct 30 19:37:37.071803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.071771    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk" pod "e32d2372-c610-4eb4-b7cd-348fd9388a07" (UID: "e32d2372-c610-4eb4-b7cd-348fd9388a07"). InnerVolumeSpecName "kube-api-access-ggxgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:37.084328 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.084257    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert" pod "e32d2372-c610-4eb4-b7cd-348fd9388a07" (UID: "e32d2372-c610-4eb4-b7cd-348fd9388a07"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:37:37.170942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.170902    5085 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e32d2372-c610-4eb4-b7cd-348fd9388a07-tls-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:37.170942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.170942    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxgk\" (UniqueName: \"kubernetes.io/projected/e32d2372-c610-4eb4-b7cd-348fd9388a07-kube-api-access-ggxgk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:37.567814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.567771    5085 generic.go:356] "Generic (PLEG): container finished" podID="e32d2372-c610-4eb4-b7cd-348fd9388a07" containerID="dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2" exitCode=0
Oct 30 19:37:37.568298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.567831    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c655d95d5-bjwvw"
Oct 30 19:37:37.568298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.567857    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" event={"ID":"e32d2372-c610-4eb4-b7cd-348fd9388a07","Type":"ContainerDied","Data":"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"}
Oct 30 19:37:37.568298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.567912    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c655d95d5-bjwvw" event={"ID":"e32d2372-c610-4eb4-b7cd-348fd9388a07","Type":"ContainerDied","Data":"c6769a4b630c515b1424c0dbba992f1ffd985e0768198826f64c8e07ca06a8fd"}
Oct 30 19:37:37.568298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.567936    5085 scope.go:122] "RemoveContainer" containerID="dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"
Oct 30 19:37:37.580396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.580369    5085 scope.go:122] "RemoveContainer" containerID="dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"
Oct 30 19:37:37.580714 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:37:37.580694    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2\": container with ID starting with dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2 not found: ID does not exist" containerID="dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"
Oct 30 19:37:37.580769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.580724    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2"} err="failed to get container status \"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2\": rpc error: code = NotFound desc = could not find container \"dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2\": container with ID starting with dafe57743e350f14f1ce6663a90479b8e46be854ef2ecab9539c2bb9aeec68a2 not found: ID does not exist"
Oct 30 19:37:37.726530 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.726493    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:37:37.731082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:37.731049    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c655d95d5-bjwvw"]
Oct 30 19:37:38.631103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:37:38.631067    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32d2372-c610-4eb4-b7cd-348fd9388a07" path="/var/lib/kubelet/pods/e32d2372-c610-4eb4-b7cd-348fd9388a07/volumes"
Oct 30 19:37:51.939124 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:37:51.939079    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:38:15.708222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.708182    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:38:15.708689 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.708674    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e32d2372-c610-4eb4-b7cd-348fd9388a07" containerName="authorino"
Oct 30 19:38:15.708689 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.708689    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32d2372-c610-4eb4-b7cd-348fd9388a07" containerName="authorino"
Oct 30 19:38:15.708787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.708777    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32d2372-c610-4eb4-b7cd-348fd9388a07" containerName="authorino"
Oct 30 19:38:15.712058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.712038    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.747826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.747793    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748025    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748076    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxkd\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748121    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748154    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748223    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748321    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748383 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748351    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.748448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.748434    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.771869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.771842    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3df7da3e-openshift-default-dockercfg-978xh\""
Oct 30 19:38:15.811948 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.811916    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:38:15.849143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849107    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849170    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849198    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849234    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxkd\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849279    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849308    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849363    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849415    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849441    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.849633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.849552    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.850057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.850031    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.850259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.850232    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.850380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.850353    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.850744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.850724    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.852172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.852155    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.852401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.852384    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.914586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.914533    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:15.914812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:15.914789    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxkd\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\") pod \"e2e-ait-3df7da3e-openshift-default-5485798879-8x262\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") " pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:16.024450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.024363    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:16.365211 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.365137    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:38:16.370023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.370003    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.432613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.430805    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f214434c-gw-openshift-default-dockercfg-sdrxb\""
Oct 30 19:38:16.435049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.435022    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:38:16.457059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457027    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457099    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457146    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457187    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457254    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457281    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xdr\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457308    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457348    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.457386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.457383    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.493306 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.493274    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:38:16.497312 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:16.497279    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26b2434_6d58_488a_b2af_13b51121fccf.slice/crio-d520701bb1bc4a8404a960ca537d134bb7b02e4c28a7173e4377d0a0aab76259 WatchSource:0}: Error finding container d520701bb1bc4a8404a960ca537d134bb7b02e4c28a7173e4377d0a0aab76259: Status 404 returned error can't find the container with id d520701bb1bc4a8404a960ca537d134bb7b02e4c28a7173e4377d0a0aab76259
Oct 30 19:38:16.499494 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.499461    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:16.499632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.499523    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:16.499632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.499554    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:16.558355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.558300    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.558487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.558389    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.558487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.558439    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.558649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.558495    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.558649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.558529    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.559389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.559290    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.559648    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.559712    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.559803    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.559831    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xdr\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.560453    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.560537    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.560605    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.561251    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.561500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.561458    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.570245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.570219    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.618817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.618746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.619169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.619145    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xdr\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\") pod \"e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") " pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.683511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.683482    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:16.797289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.797252    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerStarted","Data":"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"}
Oct 30 19:38:16.797749 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.797317    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerStarted","Data":"d520701bb1bc4a8404a960ca537d134bb7b02e4c28a7173e4377d0a0aab76259"}
Oct 30 19:38:16.912335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:16.912229    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podStartSLOduration=1.912211823 podStartE2EDuration="1.912211823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:16.911414455 +0000 UTC m=+806.415818199" watchObservedRunningTime="2026-10-30 19:38:16.912211823 +0000 UTC m=+806.416615570"
Oct 30 19:38:17.025286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.025246    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:17.031162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.031137    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:17.114298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.114268    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:38:17.115619 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:17.115588    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c3d63a_b9dc_4726_aeb9_8adb92c7695c.slice/crio-9a61a1e68f1c36a545c99d71c1c0eee154b06960d3afe274cafd854f5ff965d3 WatchSource:0}: Error finding container 9a61a1e68f1c36a545c99d71c1c0eee154b06960d3afe274cafd854f5ff965d3: Status 404 returned error can't find the container with id 9a61a1e68f1c36a545c99d71c1c0eee154b06960d3afe274cafd854f5ff965d3
Oct 30 19:38:17.118314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.118280    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:17.118425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.118361    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:17.118425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.118407    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:17.804539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.804501    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerStarted","Data":"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"}
Oct 30 19:38:17.805011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.804585    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerStarted","Data":"9a61a1e68f1c36a545c99d71c1c0eee154b06960d3afe274cafd854f5ff965d3"}
Oct 30 19:38:17.805011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.804877    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:17.805825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.805805    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:38:17.921094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:17.921036    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podStartSLOduration=1.9210221490000001 podStartE2EDuration="1.921022149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:17.91909596 +0000 UTC m=+807.423499710" watchObservedRunningTime="2026-10-30 19:38:17.921022149 +0000 UTC m=+807.425425896"
Oct 30 19:38:18.684344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:18.684313    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:18.689387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:18.689360    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:18.810901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:18.810858    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:18.811643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:18.811627    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:38:21.939089 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:38:21.939048    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:38:22.718249 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.718216    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:38:22.723156 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.723136    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.822818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.822783    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3df7da3e\""
Oct 30 19:38:22.823133 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.822890    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3df7da3e-dockercfg-qsb89\""
Oct 30 19:38:22.826903 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.826864    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:38:22.835186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.835153    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.835704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.835672    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cvb\" (UniqueName: \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.936824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.936784    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.937017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.936870    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cvb\" (UniqueName: \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.937595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.937555    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:22.996147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:22.996074    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cvb\" (UniqueName: \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\") pod \"payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:23.035854 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.035820    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:23.251435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.251357    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:38:23.256174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.256152    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.311982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.311954    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-be15b010-openshift-default-dockercfg-h2pj7\""
Oct 30 19:38:23.319418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.319383    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:38:23.341064 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341029    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341076    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341103    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341226    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqlt\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341262    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341287    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341343    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341433    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.341472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.341469    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.422785 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.422755    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:38:23.424078 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:23.424053    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe4b87c_5f7f_4fcc_ac52_493bd639ccfb.slice/crio-7bfe7c674bf82ae4ba1266f31b4fc0d5b64b3cc535cbca315023eee71800972a WatchSource:0}: Error finding container 7bfe7c674bf82ae4ba1266f31b4fc0d5b64b3cc535cbca315023eee71800972a: Status 404 returned error can't find the container with id 7bfe7c674bf82ae4ba1266f31b4fc0d5b64b3cc535cbca315023eee71800972a
Oct 30 19:38:23.442651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442615    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442679    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442717    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442740    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442776    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442821    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqlt\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442859    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442882    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.442932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.442918    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.443272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.443249    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.443503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.443481    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.443826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.443799    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.444049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.444028    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.444049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.444041    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.445844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.445809    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.445975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.445952    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.512671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.512613    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.512671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.512653    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqlt\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\") pod \"e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") " pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.569781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.569745    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:23.845294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.845206    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" event={"ID":"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb","Type":"ContainerStarted","Data":"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"}
Oct 30 19:38:23.845436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.845295    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" event={"ID":"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb","Type":"ContainerStarted","Data":"7bfe7c674bf82ae4ba1266f31b4fc0d5b64b3cc535cbca315023eee71800972a"}
Oct 30 19:38:23.845480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.845454    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:23.957199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.957147    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" podStartSLOduration=1.957133714 podStartE2EDuration="1.957133714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:23.956411492 +0000 UTC m=+813.460815241" watchObservedRunningTime="2026-10-30 19:38:23.957133714 +0000 UTC m=+813.461537502"
Oct 30 19:38:23.991044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.991019    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:38:23.992338 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:23.992311    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4212db21_597f_4e1c_a535_53eeadc3c324.slice/crio-fabf182236a73258a4a24b0a225ac50df490a1640abc2fb4582e1ddf5bdb33ed WatchSource:0}: Error finding container fabf182236a73258a4a24b0a225ac50df490a1640abc2fb4582e1ddf5bdb33ed: Status 404 returned error can't find the container with id fabf182236a73258a4a24b0a225ac50df490a1640abc2fb4582e1ddf5bdb33ed
Oct 30 19:38:23.994537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.994506    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:23.994644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.994600    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:23.994644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:23.994634    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:38:24.852489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:24.852452    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerStarted","Data":"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"}
Oct 30 19:38:24.852944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:24.852507    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerStarted","Data":"fabf182236a73258a4a24b0a225ac50df490a1640abc2fb4582e1ddf5bdb33ed"}
Oct 30 19:38:24.973830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:24.973777    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podStartSLOduration=1.973763194 podStartE2EDuration="1.973763194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:24.969853558 +0000 UTC m=+814.474257306" watchObservedRunningTime="2026-10-30 19:38:24.973763194 +0000 UTC m=+814.478166940"
Oct 30 19:38:25.570345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:25.570312    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:25.575402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:25.575375    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:25.859240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:25.859165    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:25.860070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:25.860049    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:38:27.784529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.784496    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:38:27.788433 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.788413    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:27.859664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.859305    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:38:27.887035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.886999    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncmcm\" (UniqueName: \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:27.887204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.887175    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:27.895463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.895435    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-87e6ae88\""
Oct 30 19:38:27.895630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.895504    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-87e6ae88-dockercfg-7ghd4\""
Oct 30 19:38:27.898913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.898737    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:38:27.902890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.902867    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:27.963258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.963226    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:38:27.987890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.987857    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncmcm\" (UniqueName: \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:27.988055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.987910    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmbf\" (UniqueName: \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:27.988118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.988076    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:27.988118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.988109    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:27.988795 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:27.988775    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:28.046825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.046752    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncmcm\" (UniqueName: \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\") pod \"payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:28.089238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.089200    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmbf\" (UniqueName: \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.089374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.089328    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.090121 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.090099    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.101712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.101686    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:28.150107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.150068    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmbf\" (UniqueName: \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\") pod \"payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") " pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.216173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.216138    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.470453 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.470430    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:38:28.471763 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:28.471732    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce121a0_f57e_4276_ae03_fae7514b5ab5.slice/crio-aa2b217c6deffe6f849c63fd87eb06174df1e1cbc1564273b3f39325693e71f1 WatchSource:0}: Error finding container aa2b217c6deffe6f849c63fd87eb06174df1e1cbc1564273b3f39325693e71f1: Status 404 returned error can't find the container with id aa2b217c6deffe6f849c63fd87eb06174df1e1cbc1564273b3f39325693e71f1
Oct 30 19:38:28.595001 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.594977    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:38:28.595946 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:28.595914    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c3cc80_c6c5_400c_8970_0a8da3dc66c6.slice/crio-58fac04039cd91b636406ba12bd9f5357103c6ac9ece9ec024d1e6d20eee53dd WatchSource:0}: Error finding container 58fac04039cd91b636406ba12bd9f5357103c6ac9ece9ec024d1e6d20eee53dd: Status 404 returned error can't find the container with id 58fac04039cd91b636406ba12bd9f5357103c6ac9ece9ec024d1e6d20eee53dd
Oct 30 19:38:28.890628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.890525    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" event={"ID":"79c3cc80-c6c5-400c-8970-0a8da3dc66c6","Type":"ContainerStarted","Data":"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"}
Oct 30 19:38:28.890628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.890603    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" event={"ID":"79c3cc80-c6c5-400c-8970-0a8da3dc66c6","Type":"ContainerStarted","Data":"58fac04039cd91b636406ba12bd9f5357103c6ac9ece9ec024d1e6d20eee53dd"}
Oct 30 19:38:28.891102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.890747    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:28.892157 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.892134    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" event={"ID":"1ce121a0-f57e-4276-ae03-fae7514b5ab5","Type":"ContainerStarted","Data":"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"}
Oct 30 19:38:28.892286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.892173    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" event={"ID":"1ce121a0-f57e-4276-ae03-fae7514b5ab5","Type":"ContainerStarted","Data":"aa2b217c6deffe6f849c63fd87eb06174df1e1cbc1564273b3f39325693e71f1"}
Oct 30 19:38:28.892286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:28.892249    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:29.009972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:29.009917    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" podStartSLOduration=2.009903367 podStartE2EDuration="2.009903367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:29.007335594 +0000 UTC m=+818.511739343" watchObservedRunningTime="2026-10-30 19:38:29.009903367 +0000 UTC m=+818.514307114"
Oct 30 19:38:29.118110 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:29.118055    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" podStartSLOduration=2.118041655 podStartE2EDuration="2.118041655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:29.117879451 +0000 UTC m=+818.622283212" watchObservedRunningTime="2026-10-30 19:38:29.118041655 +0000 UTC m=+818.622445402"
Oct 30 19:38:33.149737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.149705    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:38:33.153379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.153360    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.209011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.208980    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:38:33.259450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.255925    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5442f600-dockercfg-96z8p\""
Oct 30 19:38:33.259450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.256015    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5442f600\""
Oct 30 19:38:33.259450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.257980    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.259450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.258136    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7g9\" (UniqueName: \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.307597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.307553    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:38:33.311327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.311306    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.358746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.358713    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7g9\" (UniqueName: \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.358914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.358806    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.359571 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.359540    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.370435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.370405    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:38:33.417233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.417167    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7g9\" (UniqueName: \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\") pod \"payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.459692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.459664    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg26h\" (UniqueName: \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.459842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.459791    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.467515 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.467495    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.561347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.561316    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.561529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.561401    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg26h\" (UniqueName: \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.562181 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.562156    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.620148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.620119    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg26h\" (UniqueName: \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\") pod \"payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") " pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.623959 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.623940    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:33.846538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.846515    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:38:33.848493 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:33.848460    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1f77c0_5afd_420c_8998_dcd18ba70265.slice/crio-ad031df58b2529a59c04a4b803b7449dd72a1bfa09d0d5320ac3a352a865e666 WatchSource:0}: Error finding container ad031df58b2529a59c04a4b803b7449dd72a1bfa09d0d5320ac3a352a865e666: Status 404 returned error can't find the container with id ad031df58b2529a59c04a4b803b7449dd72a1bfa09d0d5320ac3a352a865e666
Oct 30 19:38:33.923243 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.923213    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" event={"ID":"4a1f77c0-5afd-420c-8998-dcd18ba70265","Type":"ContainerStarted","Data":"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"}
Oct 30 19:38:33.923380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.923267    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" event={"ID":"4a1f77c0-5afd-420c-8998-dcd18ba70265","Type":"ContainerStarted","Data":"ad031df58b2529a59c04a4b803b7449dd72a1bfa09d0d5320ac3a352a865e666"}
Oct 30 19:38:33.923463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.923381    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:33.992961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:33.992940    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:38:33.994608 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:38:33.994576    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5880ef95_c031_4e84_8124_35d8a8c2701d.slice/crio-0da8b4f112738f0475bb5f0d8ab09a13fec6fcf8de38a25c47ee0294bc2a908c WatchSource:0}: Error finding container 0da8b4f112738f0475bb5f0d8ab09a13fec6fcf8de38a25c47ee0294bc2a908c: Status 404 returned error can't find the container with id 0da8b4f112738f0475bb5f0d8ab09a13fec6fcf8de38a25c47ee0294bc2a908c
Oct 30 19:38:34.035299 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:34.035257    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" podStartSLOduration=1.035243229 podStartE2EDuration="1.035243229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:34.033709389 +0000 UTC m=+823.538113140" watchObservedRunningTime="2026-10-30 19:38:34.035243229 +0000 UTC m=+823.539646986"
Oct 30 19:38:34.856084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:34.856056    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:38:34.929970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:34.929934    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" event={"ID":"5880ef95-c031-4e84-8124-35d8a8c2701d","Type":"ContainerStarted","Data":"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"}
Oct 30 19:38:34.930144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:34.930010    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" event={"ID":"5880ef95-c031-4e84-8124-35d8a8c2701d","Type":"ContainerStarted","Data":"0da8b4f112738f0475bb5f0d8ab09a13fec6fcf8de38a25c47ee0294bc2a908c"}
Oct 30 19:38:34.930144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:34.930034    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:35.079812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:35.079755    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" podStartSLOduration=2.079741227 podStartE2EDuration="2.079741227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:35.077452191 +0000 UTC m=+824.581855937" watchObservedRunningTime="2026-10-30 19:38:35.079741227 +0000 UTC m=+824.584144974"
Oct 30 19:38:38.723164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:38.723131    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:38:39.900944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:39.900915    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:38:39.901318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:39.900980    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:38:44.933070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:44.933036    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:38:45.939035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:45.939009    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:38:49.934746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:38:49.934706    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:38:51.938734 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:38:51.938697    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:39:01.117065 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.117033    5085 generic.go:356] "Generic (PLEG): container finished" podID="c26b2434-6d58-488a-b2af-13b51121fccf" containerID="37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927" exitCode=0
Oct 30 19:39:01.117674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.117237    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerDied","Data":"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"}
Oct 30 19:39:01.117876 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.117856    5085 scope.go:122] "RemoveContainer" containerID="37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"
Oct 30 19:39:01.119189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.119157    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:01.119279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.119227    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:01.119279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.119260    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:01.834859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.834805    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"]
Oct 30 19:39:01.842940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.842890    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:01.893585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.893538    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:01.893782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.893756    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:39:01.921260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:01.921221    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"]
Oct 30 19:39:02.031587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.028120    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.031587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.028260    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dts82\" (UniqueName: \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.031587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.028330    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.135454 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.135364    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dts82\" (UniqueName: \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.135865 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.135460    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.135865 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.135535    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.136399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.136373    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.142222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.142189    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.148399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.148372    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerStarted","Data":"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"}
Oct 30 19:39:02.224587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.224283    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dts82\" (UniqueName: \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\") pod \"maas-api-revoke-keys-e2e-ait-87e6ae88-52nph\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.459751 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.459715    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:02.828586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:02.807711    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:03.037104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:03.036673    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:03.037104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:03.036767    5085 patch_prober.go:29] interesting pod/e2e-ait-3df7da3e-openshift-default-5485798879-8x262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:39:03.037104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:03.036829    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 19:39:03.037104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:03.037008    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"]
Oct 30 19:39:03.178587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:03.176625    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph" event={"ID":"2c82200c-2324-4027-ad4b-4a6be44c72e3","Type":"ContainerStarted","Data":"fb063814f9b50ffb27c37fba61e4ce91448a220dddfaf134da314d0e41ac5e04"}
Oct 30 19:39:04.030588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:04.027128    5085 patch_prober.go:29] interesting pod/e2e-ait-3df7da3e-openshift-default-5485798879-8x262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:39:04.030588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:04.027191    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 19:39:04.851101 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:04.851019    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:39:05.025265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:05.025228    5085 patch_prober.go:29] interesting pod/e2e-ait-3df7da3e-openshift-default-5485798879-8x262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:39:05.025430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:05.025279    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 19:39:06.025179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:06.025145    5085 patch_prober.go:29] interesting pod/e2e-ait-3df7da3e-openshift-default-5485798879-8x262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:39:06.025619 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:06.025212    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 19:39:07.024921 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:07.024834    5085 patch_prober.go:29] interesting pod/e2e-ait-3df7da3e-openshift-default-5485798879-8x262 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:39:07.024921 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:07.024902    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 19:39:08.029912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:08.029889    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:08.031323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:08.031297    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:09.231073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.231037    5085 generic.go:356] "Generic (PLEG): container finished" podID="2c82200c-2324-4027-ad4b-4a6be44c72e3" containerID="508a15966ab7389e84756b87aaf653e7ad46d8c011cf03026512b1a3253d69c9" exitCode=0
Oct 30 19:39:09.231540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.231129    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph" event={"ID":"2c82200c-2324-4027-ad4b-4a6be44c72e3","Type":"ContainerDied","Data":"508a15966ab7389e84756b87aaf653e7ad46d8c011cf03026512b1a3253d69c9"}
Oct 30 19:39:09.232720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.232695    5085 generic.go:356] "Generic (PLEG): container finished" podID="4212db21-597f-4e1c-a535-53eeadc3c324" containerID="2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6" exitCode=0
Oct 30 19:39:09.232819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.232767    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerDied","Data":"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"}
Oct 30 19:39:09.233124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.233109    5085 scope.go:122] "RemoveContainer" containerID="2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"
Oct 30 19:39:09.234333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234313    5085 generic.go:356] "Generic (PLEG): container finished" podID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerID="9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908" exitCode=0
Oct 30 19:39:09.234432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234329    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.234432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234387    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.234432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234390    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerDied","Data":"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"}
Oct 30 19:39:09.234432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234418    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.234882 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.234788    5085 scope.go:122] "RemoveContainer" containerID="9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"
Oct 30 19:39:09.235980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.235957    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.236042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.236015    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.236083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.236047    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:09.330085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.330060    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"]
Oct 30 19:39:09.335167 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.335004    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.378678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.378637    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"]
Oct 30 19:39:09.437615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.436094    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.437615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.436158    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.437615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.436280    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9n54\" (UniqueName: \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.537467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.537391    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.537467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.537434    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.537748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.537473    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9n54\" (UniqueName: \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.538350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.538322    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.539797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.539763    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.595874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.595843    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9n54\" (UniqueName: \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\") pod \"maas-api-revoke-keys-e2e-ait-5442f600-ffgj9\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:09.652273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:09.652234    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:10.050633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.050608    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"]
Oct 30 19:39:10.052107 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:39:10.052083    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7113e2de_8a0e_4996_b7dd_48eb42d30c11.slice/crio-189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0 WatchSource:0}: Error finding container 189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0: Status 404 returned error can't find the container with id 189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0
Oct 30 19:39:10.242148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.242112    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9" event={"ID":"7113e2de-8a0e-4996-b7dd-48eb42d30c11","Type":"ContainerStarted","Data":"189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0"}
Oct 30 19:39:10.244069 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.244041    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerStarted","Data":"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"}
Oct 30 19:39:10.246102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.246081    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerStarted","Data":"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"}
Oct 30 19:39:10.547495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.547467    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:10.595588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.584714    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:39:10.595588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.586038    5085 patch_prober.go:29] interesting pod/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:39:10.595588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.586107    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:39:10.647618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.647532    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\" (UniqueName: \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\") pod \"2c82200c-2324-4027-ad4b-4a6be44c72e3\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") "
Oct 30 19:39:10.647618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.647581    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\") pod \"2c82200c-2324-4027-ad4b-4a6be44c72e3\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") "
Oct 30 19:39:10.647808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.647661    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\") pod \"2c82200c-2324-4027-ad4b-4a6be44c72e3\" (UID: \"2c82200c-2324-4027-ad4b-4a6be44c72e3\") "
Oct 30 19:39:10.648187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.648158    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca" pod "2c82200c-2324-4027-ad4b-4a6be44c72e3" (UID: "2c82200c-2324-4027-ad4b-4a6be44c72e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:10.650016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.649991    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca" pod "2c82200c-2324-4027-ad4b-4a6be44c72e3" (UID: "2c82200c-2324-4027-ad4b-4a6be44c72e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:10.650231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.650206    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82" pod "2c82200c-2324-4027-ad4b-4a6be44c72e3" (UID: "2c82200c-2324-4027-ad4b-4a6be44c72e3"). InnerVolumeSpecName "kube-api-access-dts82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:10.720673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.720635    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:39:10.720911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.720889    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:39:10.721259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.721235    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:39:10.749577 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.749536    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dts82\" (UniqueName: \"kubernetes.io/projected/2c82200c-2324-4027-ad4b-4a6be44c72e3-kube-api-access-dts82\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:10.749740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.749726    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:10.749803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.749796    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c82200c-2324-4027-ad4b-4a6be44c72e3-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:10.868588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:10.861033    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:39:11.254325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.254291    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph" event={"ID":"2c82200c-2324-4027-ad4b-4a6be44c72e3","Type":"ContainerDied","Data":"fb063814f9b50ffb27c37fba61e4ce91448a220dddfaf134da314d0e41ac5e04"}
Oct 30 19:39:11.254728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.254340    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb063814f9b50ffb27c37fba61e4ce91448a220dddfaf134da314d0e41ac5e04"
Oct 30 19:39:11.254728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.254391    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"
Oct 30 19:39:11.255916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.255892    5085 generic.go:356] "Generic (PLEG): container finished" podID="7113e2de-8a0e-4996-b7dd-48eb42d30c11" containerID="0b3074408e32048271da551ed566cf0d04c8ac8e093ab2c0c423fc80201bcd75" exitCode=0
Oct 30 19:39:11.267612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.256669    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9" event={"ID":"7113e2de-8a0e-4996-b7dd-48eb42d30c11","Type":"ContainerDied","Data":"0b3074408e32048271da551ed566cf0d04c8ac8e093ab2c0c423fc80201bcd75"}
Oct 30 19:39:11.573644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.570953    5085 patch_prober.go:29] interesting pod/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:39:11.573644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.571023    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:39:11.702587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.701950    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:39:11.702587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:11.702010    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:39:12.323720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.323688    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:39:12.441775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.441743    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:39:12.546000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.545975    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:12.587650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.584059    5085 patch_prober.go:29] interesting pod/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:39:12.587650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.584132    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.689166    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.689219    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.719073    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\") pod \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") "
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.719617    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\") pod \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") "
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.720033    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca" pod "7113e2de-8a0e-4996-b7dd-48eb42d30c11" (UID: "7113e2de-8a0e-4996-b7dd-48eb42d30c11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.720125    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\" (UniqueName: \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\") pod \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\" (UID: \"7113e2de-8a0e-4996-b7dd-48eb42d30c11\") "
Oct 30 19:39:12.721586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.720975    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.722709 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.722609    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca" pod "7113e2de-8a0e-4996-b7dd-48eb42d30c11" (UID: "7113e2de-8a0e-4996-b7dd-48eb42d30c11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:12.722709 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.722655    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54" pod "7113e2de-8a0e-4996-b7dd-48eb42d30c11" (UID: "7113e2de-8a0e-4996-b7dd-48eb42d30c11"). InnerVolumeSpecName "kube-api-access-p9n54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:12.825150 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.825113    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9n54\" (UniqueName: \"kubernetes.io/projected/7113e2de-8a0e-4996-b7dd-48eb42d30c11-kube-api-access-p9n54\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:12.825150 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:12.825147    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7113e2de-8a0e-4996-b7dd-48eb42d30c11-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:13.317336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.317300    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9" event={"ID":"7113e2de-8a0e-4996-b7dd-48eb42d30c11","Type":"ContainerDied","Data":"189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0"}
Oct 30 19:39:13.317486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.317348    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="189daa6a109d121232fcaa0a8bf1e099b3d2bc5bbffaed5d4ae15d36ec6d35d0"
Oct 30 19:39:13.317486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.317358    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"
Oct 30 19:39:13.607690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.580005    5085 patch_prober.go:29] interesting pod/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:39:13.607690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.580069    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:39:13.684123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.684001    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:39:13.684123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:13.684063    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:39:14.579587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:14.577090    5085 patch_prober.go:29] interesting pod/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:39:14.579587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:14.577154    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 30 19:39:14.687045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:14.687005    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 30 19:39:14.687683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:14.687641    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 30 19:39:15.574902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:15.574872    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:39:15.576007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:15.575984    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:39:15.688517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:15.688487    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:39:15.688915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:15.688788    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:39:15.689355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:15.689337    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:39:17.325310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.325267    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" podUID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" containerName="payload-processing" containerID="cri-o://7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea" gracePeriod=25
Oct 30 19:39:17.442450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.442412    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" podUID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" containerName="payload-pre-processing" containerID="cri-o://30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22" gracePeriod=25
Oct 30 19:39:17.599812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.599786    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:39:17.722358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.722333    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:39:17.774813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.774779    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\") pod \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") "
Oct 30 19:39:17.774813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.774818    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\" (UniqueName: \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\") pod \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\" (UID: \"79c3cc80-c6c5-400c-8970-0a8da3dc66c6\") "
Oct 30 19:39:17.775045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.774962    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\") pod \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") "
Oct 30 19:39:17.775045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.775028    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\" (UniqueName: \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\") pod \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\" (UID: \"1ce121a0-f57e-4276-ae03-fae7514b5ab5\") "
Oct 30 19:39:17.775640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.775550    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume" pod "79c3cc80-c6c5-400c-8970-0a8da3dc66c6" (UID: "79c3cc80-c6c5-400c-8970-0a8da3dc66c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:17.776155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.776049    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume" pod "1ce121a0-f57e-4276-ae03-fae7514b5ab5" (UID: "1ce121a0-f57e-4276-ae03-fae7514b5ab5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:17.784413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.776588    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce121a0-f57e-4276-ae03-fae7514b5ab5-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:17.784413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.776614    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:17.784413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.778078    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf" pod "79c3cc80-c6c5-400c-8970-0a8da3dc66c6" (UID: "79c3cc80-c6c5-400c-8970-0a8da3dc66c6"). InnerVolumeSpecName "kube-api-access-qkmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:17.786067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.786042    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm" pod "1ce121a0-f57e-4276-ae03-fae7514b5ab5" (UID: "1ce121a0-f57e-4276-ae03-fae7514b5ab5"). InnerVolumeSpecName "kube-api-access-ncmcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:17.878198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.878115    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncmcm\" (UniqueName: \"kubernetes.io/projected/1ce121a0-f57e-4276-ae03-fae7514b5ab5-kube-api-access-ncmcm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:17.878198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:17.878145    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmbf\" (UniqueName: \"kubernetes.io/projected/79c3cc80-c6c5-400c-8970-0a8da3dc66c6-kube-api-access-qkmbf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:18.355822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.355792    5085 generic.go:356] "Generic (PLEG): container finished" podID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" containerID="7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea" exitCode=0
Oct 30 19:39:18.356195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.355851    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"
Oct 30 19:39:18.356195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.355876    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" event={"ID":"79c3cc80-c6c5-400c-8970-0a8da3dc66c6","Type":"ContainerDied","Data":"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"}
Oct 30 19:39:18.356195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.355962    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn" event={"ID":"79c3cc80-c6c5-400c-8970-0a8da3dc66c6","Type":"ContainerDied","Data":"58fac04039cd91b636406ba12bd9f5357103c6ac9ece9ec024d1e6d20eee53dd"}
Oct 30 19:39:18.356195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.355991    5085 scope.go:122] "RemoveContainer" containerID="7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"
Oct 30 19:39:18.357313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.357288    5085 generic.go:356] "Generic (PLEG): container finished" podID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" containerID="30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22" exitCode=0
Oct 30 19:39:18.357372 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.357360    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"
Oct 30 19:39:18.357417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.357371    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" event={"ID":"1ce121a0-f57e-4276-ae03-fae7514b5ab5","Type":"ContainerDied","Data":"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"}
Oct 30 19:39:18.357466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.357426    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh" event={"ID":"1ce121a0-f57e-4276-ae03-fae7514b5ab5","Type":"ContainerDied","Data":"aa2b217c6deffe6f849c63fd87eb06174df1e1cbc1564273b3f39325693e71f1"}
Oct 30 19:39:18.366703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.366685    5085 scope.go:122] "RemoveContainer" containerID="7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"
Oct 30 19:39:18.366996 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:18.366976    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea\": container with ID starting with 7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea not found: ID does not exist" containerID="7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"
Oct 30 19:39:18.367082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.367006    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea"} err="failed to get container status \"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea\": rpc error: code = NotFound desc = could not find container \"7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea\": container with ID starting with 7535336e1541c54498b819d976582c3c2275579c7daaf3a02fc8481c9fe3f0ea not found: ID does not exist"
Oct 30 19:39:18.367082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.367030    5085 scope.go:122] "RemoveContainer" containerID="30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"
Oct 30 19:39:18.376183 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.376166    5085 scope.go:122] "RemoveContainer" containerID="30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"
Oct 30 19:39:18.376431 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:18.376412    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22\": container with ID starting with 30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22 not found: ID does not exist" containerID="30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"
Oct 30 19:39:18.376487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.376441    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22"} err="failed to get container status \"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22\": rpc error: code = NotFound desc = could not find container \"30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22\": container with ID starting with 30992842ff835d96f3ee4961a5da4d9e6dceeb7f5d739c26d6cf6f11383bfb22 not found: ID does not exist"
Oct 30 19:39:18.511680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.511650    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:39:18.553512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.553466    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-87e6ae88-dd69676cf-h2jvh"]
Oct 30 19:39:18.621738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.621597    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:39:18.630825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.630795    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" path="/var/lib/kubelet/pods/1ce121a0-f57e-4276-ae03-fae7514b5ab5/volumes"
Oct 30 19:39:18.631119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:18.631104    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-87e6ae88-65fb98d7bb-55ggn"]
Oct 30 19:39:20.310073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:20.310041    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:39:20.419844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:20.419815    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:39:20.633295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:20.633216    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" path="/var/lib/kubelet/pods/79c3cc80-c6c5-400c-8970-0a8da3dc66c6/volumes"
Oct 30 19:39:21.347840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.347806    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"]
Oct 30 19:39:21.348321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348307    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2c82200c-2324-4027-ad4b-4a6be44c72e3" containerName="revoke-keys"
Oct 30 19:39:21.348360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348322    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c82200c-2324-4027-ad4b-4a6be44c72e3" containerName="revoke-keys"
Oct 30 19:39:21.348360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348345    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" containerName="payload-pre-processing"
Oct 30 19:39:21.348360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348350    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" containerName="payload-pre-processing"
Oct 30 19:39:21.348360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348360    5085 cpu_manager.go:414] "containerMap: removing container" podUID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" containerName="payload-processing"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348365    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" containerName="payload-processing"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348373    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7113e2de-8a0e-4996-b7dd-48eb42d30c11" containerName="revoke-keys"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348377    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7113e2de-8a0e-4996-b7dd-48eb42d30c11" containerName="revoke-keys"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348447    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce121a0-f57e-4276-ae03-fae7514b5ab5" containerName="payload-pre-processing"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348454    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c3cc80-c6c5-400c-8970-0a8da3dc66c6" containerName="payload-processing"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348460    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7113e2de-8a0e-4996-b7dd-48eb42d30c11" containerName="revoke-keys"
Oct 30 19:39:21.348507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.348470    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c82200c-2324-4027-ad4b-4a6be44c72e3" containerName="revoke-keys"
Oct 30 19:39:21.353227 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.353211    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.403541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.403509    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:21.404139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.404117    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:39:21.417686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.417651    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6llr\" (UniqueName: \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.417864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.417848    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.417985 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.417965    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.419333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.419305    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"]
Oct 30 19:39:21.519367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.519326    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.519597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.519424    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.519597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.519449    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6llr\" (UniqueName: \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.520030 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.520004    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.521814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.521789    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.578018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.577984    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6llr\" (UniqueName: \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\") pod \"maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.667600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:21.667486    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:21.939714 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:21.939676    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:39:22.037375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:22.037349    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"]
Oct 30 19:39:22.039343 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:39:22.039307    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9869b5f1_928c_4765_b944_dd85f0f82d76.slice/crio-b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4 WatchSource:0}: Error finding container b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4: Status 404 returned error can't find the container with id b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4
Oct 30 19:39:22.386583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:22.386527    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz" event={"ID":"9869b5f1-928c-4765-b944-dd85f0f82d76","Type":"ContainerStarted","Data":"4f269dc888c5f9b16c2b03da41d54a94bed122b795ea67d07795f65bf0b62a96"}
Oct 30 19:39:22.386908 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:22.386592    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz" event={"ID":"9869b5f1-928c-4765-b944-dd85f0f82d76","Type":"ContainerStarted","Data":"b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4"}
Oct 30 19:39:22.497554 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:22.497495    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz" podStartSLOduration=1.497479115 podStartE2EDuration="1.497479115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:22.496299024 +0000 UTC m=+872.000702772" watchObservedRunningTime="2026-10-30 19:39:22.497479115 +0000 UTC m=+872.001882862"
Oct 30 19:39:22.962490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:22.962447    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:39:23.402132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:23.402088    5085 generic.go:356] "Generic (PLEG): container finished" podID="9869b5f1-928c-4765-b944-dd85f0f82d76" containerID="4f269dc888c5f9b16c2b03da41d54a94bed122b795ea67d07795f65bf0b62a96" exitCode=0
Oct 30 19:39:23.402726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:23.402169    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz" event={"ID":"9869b5f1-928c-4765-b944-dd85f0f82d76","Type":"ContainerDied","Data":"4f269dc888c5f9b16c2b03da41d54a94bed122b795ea67d07795f65bf0b62a96"}
Oct 30 19:39:24.591200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.591172    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:24.652066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.652029    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\" (UniqueName: \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\") pod \"9869b5f1-928c-4765-b944-dd85f0f82d76\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") "
Oct 30 19:39:24.652242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.652093    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\") pod \"9869b5f1-928c-4765-b944-dd85f0f82d76\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") "
Oct 30 19:39:24.652242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.652166    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\") pod \"9869b5f1-928c-4765-b944-dd85f0f82d76\" (UID: \"9869b5f1-928c-4765-b944-dd85f0f82d76\") "
Oct 30 19:39:24.652595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.652541    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca" pod "9869b5f1-928c-4765-b944-dd85f0f82d76" (UID: "9869b5f1-928c-4765-b944-dd85f0f82d76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:24.654754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.654728    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca" pod "9869b5f1-928c-4765-b944-dd85f0f82d76" (UID: "9869b5f1-928c-4765-b944-dd85f0f82d76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:24.654829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.654734    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr" pod "9869b5f1-928c-4765-b944-dd85f0f82d76" (UID: "9869b5f1-928c-4765-b944-dd85f0f82d76"). InnerVolumeSpecName "kube-api-access-j6llr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:24.754128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.754096    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:24.754128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.754125    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9869b5f1-928c-4765-b944-dd85f0f82d76-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:24.754128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.754136    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6llr\" (UniqueName: \"kubernetes.io/projected/9869b5f1-928c-4765-b944-dd85f0f82d76-kube-api-access-j6llr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:24.927275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.927237    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"]
Oct 30 19:39:24.936525 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:24.936491    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-87e6ae88-52nph"]
Oct 30 19:39:25.311445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.311321    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" podUID="5880ef95-c031-4e84-8124-35d8a8c2701d" containerName="payload-processing" containerID="cri-o://feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0" gracePeriod=25
Oct 30 19:39:25.418445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.418414    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"
Oct 30 19:39:25.418445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.418437    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz" event={"ID":"9869b5f1-928c-4765-b944-dd85f0f82d76","Type":"ContainerDied","Data":"b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4"}
Oct 30 19:39:25.418666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.418503    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b836d5ca121de9ae6017bb31c82cd67404c6a88f8467bac2b4acdcb53aabb1d4"
Oct 30 19:39:25.420290 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.420260    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" podUID="4a1f77c0-5afd-420c-8998-dcd18ba70265" containerName="payload-pre-processing" containerID="cri-o://fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31" gracePeriod=25
Oct 30 19:39:25.586337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.586310    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:39:25.663991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.663952    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\" (UniqueName: \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\") pod \"5880ef95-c031-4e84-8124-35d8a8c2701d\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") "
Oct 30 19:39:25.664367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.664005    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\") pod \"5880ef95-c031-4e84-8124-35d8a8c2701d\" (UID: \"5880ef95-c031-4e84-8124-35d8a8c2701d\") "
Oct 30 19:39:25.664428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.664390    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume" pod "5880ef95-c031-4e84-8124-35d8a8c2701d" (UID: "5880ef95-c031-4e84-8124-35d8a8c2701d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:25.666127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.666101    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h" pod "5880ef95-c031-4e84-8124-35d8a8c2701d" (UID: "5880ef95-c031-4e84-8124-35d8a8c2701d"). InnerVolumeSpecName "kube-api-access-xg26h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:25.709852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.709815    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:39:25.765325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.765289    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\") pod \"4a1f77c0-5afd-420c-8998-dcd18ba70265\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") "
Oct 30 19:39:25.765541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.765380    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\" (UniqueName: \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\") pod \"4a1f77c0-5afd-420c-8998-dcd18ba70265\" (UID: \"4a1f77c0-5afd-420c-8998-dcd18ba70265\") "
Oct 30 19:39:25.765696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.765673    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume" pod "4a1f77c0-5afd-420c-8998-dcd18ba70265" (UID: "4a1f77c0-5afd-420c-8998-dcd18ba70265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:25.765771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.765684    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg26h\" (UniqueName: \"kubernetes.io/projected/5880ef95-c031-4e84-8124-35d8a8c2701d-kube-api-access-xg26h\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.765771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.765719    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5880ef95-c031-4e84-8124-35d8a8c2701d-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.767501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.767470    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9" pod "4a1f77c0-5afd-420c-8998-dcd18ba70265" (UID: "4a1f77c0-5afd-420c-8998-dcd18ba70265"). InnerVolumeSpecName "kube-api-access-fs7g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:25.866471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.866390    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1f77c0-5afd-420c-8998-dcd18ba70265-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:25.866471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:25.866420    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7g9\" (UniqueName: \"kubernetes.io/projected/4a1f77c0-5afd-420c-8998-dcd18ba70265-kube-api-access-fs7g9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:26.020956 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.020913    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"]
Oct 30 19:39:26.023699 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.023675    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5442f600-ffgj9"]
Oct 30 19:39:26.427230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.427191    5085 generic.go:356] "Generic (PLEG): container finished" podID="5880ef95-c031-4e84-8124-35d8a8c2701d" containerID="feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0" exitCode=0
Oct 30 19:39:26.427455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.427245    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" event={"ID":"5880ef95-c031-4e84-8124-35d8a8c2701d","Type":"ContainerDied","Data":"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"}
Oct 30 19:39:26.427455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.427264    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"
Oct 30 19:39:26.427455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.427309    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq" event={"ID":"5880ef95-c031-4e84-8124-35d8a8c2701d","Type":"ContainerDied","Data":"0da8b4f112738f0475bb5f0d8ab09a13fec6fcf8de38a25c47ee0294bc2a908c"}
Oct 30 19:39:26.427455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.427335    5085 scope.go:122] "RemoveContainer" containerID="feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"
Oct 30 19:39:26.433829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.433799    5085 generic.go:356] "Generic (PLEG): container finished" podID="4a1f77c0-5afd-420c-8998-dcd18ba70265" containerID="fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31" exitCode=0
Oct 30 19:39:26.433968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.433848    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"
Oct 30 19:39:26.433968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.433878    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" event={"ID":"4a1f77c0-5afd-420c-8998-dcd18ba70265","Type":"ContainerDied","Data":"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"}
Oct 30 19:39:26.433968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.433922    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj" event={"ID":"4a1f77c0-5afd-420c-8998-dcd18ba70265","Type":"ContainerDied","Data":"ad031df58b2529a59c04a4b803b7449dd72a1bfa09d0d5320ac3a352a865e666"}
Oct 30 19:39:26.440765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.440740    5085 scope.go:122] "RemoveContainer" containerID="feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"
Oct 30 19:39:26.441096 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:26.441075    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0\": container with ID starting with feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0 not found: ID does not exist" containerID="feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"
Oct 30 19:39:26.441178 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.441105    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0"} err="failed to get container status \"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0\": rpc error: code = NotFound desc = could not find container \"feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0\": container with ID starting with feca4823b9147cd3817e10d1dc28d58476509222ea83a7ee682847de99a7a6b0 not found: ID does not exist"
Oct 30 19:39:26.441178 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.441127    5085 scope.go:122] "RemoveContainer" containerID="fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"
Oct 30 19:39:26.451520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.451492    5085 scope.go:122] "RemoveContainer" containerID="fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"
Oct 30 19:39:26.451859 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:26.451837    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31\": container with ID starting with fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31 not found: ID does not exist" containerID="fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"
Oct 30 19:39:26.451930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.451892    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31"} err="failed to get container status \"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31\": rpc error: code = NotFound desc = could not find container \"fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31\": container with ID starting with fa719449f03263db2f79b40716dc2093f5433716967206e3f5fb3da575023b31 not found: ID does not exist"
Oct 30 19:39:26.603224 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.603189    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:39:26.607162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.607127    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5442f600-85fc7c58b7-nvtwj"]
Oct 30 19:39:26.632115 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.632077    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c82200c-2324-4027-ad4b-4a6be44c72e3" path="/var/lib/kubelet/pods/2c82200c-2324-4027-ad4b-4a6be44c72e3/volumes"
Oct 30 19:39:26.632504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.632486    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1f77c0-5afd-420c-8998-dcd18ba70265" path="/var/lib/kubelet/pods/4a1f77c0-5afd-420c-8998-dcd18ba70265/volumes"
Oct 30 19:39:26.632904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.632888    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7113e2de-8a0e-4996-b7dd-48eb42d30c11" path="/var/lib/kubelet/pods/7113e2de-8a0e-4996-b7dd-48eb42d30c11/volumes"
Oct 30 19:39:26.701629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.701518    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:39:26.726297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:26.726253    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5442f600-579bc8b864-jmxbq"]
Oct 30 19:39:27.831334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831300    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 19:39:27.831819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831773    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4a1f77c0-5afd-420c-8998-dcd18ba70265" containerName="payload-pre-processing"
Oct 30 19:39:27.831819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831785    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1f77c0-5afd-420c-8998-dcd18ba70265" containerName="payload-pre-processing"
Oct 30 19:39:27.831819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831816    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5880ef95-c031-4e84-8124-35d8a8c2701d" containerName="payload-processing"
Oct 30 19:39:27.831921 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831821    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5880ef95-c031-4e84-8124-35d8a8c2701d" containerName="payload-processing"
Oct 30 19:39:27.831921 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831843    5085 cpu_manager.go:414] "containerMap: removing container" podUID="9869b5f1-928c-4765-b944-dd85f0f82d76" containerName="revoke-keys"
Oct 30 19:39:27.831921 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831850    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9869b5f1-928c-4765-b944-dd85f0f82d76" containerName="revoke-keys"
Oct 30 19:39:27.832010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831924    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5880ef95-c031-4e84-8124-35d8a8c2701d" containerName="payload-processing"
Oct 30 19:39:27.832010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831939    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1f77c0-5afd-420c-8998-dcd18ba70265" containerName="payload-pre-processing"
Oct 30 19:39:27.832010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.831947    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="9869b5f1-928c-4765-b944-dd85f0f82d76" containerName="revoke-keys"
Oct 30 19:39:27.836344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.836325    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887739    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887791    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887833    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgdf\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887862    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887886    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887924    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.887975    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.888005    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.888075    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.889585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.889540    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ec609-openshift-default-dockercfg-4txpk\""
Oct 30 19:39:27.935307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.935276    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 19:39:27.963341 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.963295    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" podUID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" containerName="payload-pre-processing" containerID="cri-o://39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e" gracePeriod=25
Oct 30 19:39:27.989073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989042    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989087    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989156    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989215    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989247    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989297    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgdf\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989331    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989355    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989405    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989491    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989542    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989773 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989604    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.989831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.989808    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.990178 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.990156    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.991524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.991502    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:27.991862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:27.991843    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:28.029184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.029147    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"]
Oct 30 19:39:28.034194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.034171    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.056049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.056017    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:28.081420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.081338    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgdf\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:28.090923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.090824    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:39:28.091712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.091689    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:39:28.092866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.092428    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999hj\" (UniqueName: \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.092866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.092479    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.092866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.092521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.097397 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.097370    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"]
Oct 30 19:39:28.149423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.149390    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:28.193427 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.193385    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999hj\" (UniqueName: \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.193929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.193435    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.193929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.193472    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.194996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.194942    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.211587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.209817    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.260103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.260067    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999hj\" (UniqueName: \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\") pod \"maas-api-revoke-keys-e2e-derive-be15b010-vc9zs\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.285296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.285268    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:39:28.294278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.294253    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\") pod \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") "
Oct 30 19:39:28.294447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.294296    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\" (UniqueName: \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\") pod \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\" (UID: \"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb\") "
Oct 30 19:39:28.294661 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.294639    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume" pod "6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" (UID: "6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:28.294738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.294722    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:28.296435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.296410    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb" pod "6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" (UID: "6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb"). InnerVolumeSpecName "kube-api-access-w6cvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:28.395825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.395746    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cvb\" (UniqueName: \"kubernetes.io/projected/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb-kube-api-access-w6cvb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:28.403224 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.403201    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:28.451737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.451536    5085 generic.go:356] "Generic (PLEG): container finished" podID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" containerID="39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e" exitCode=0
Oct 30 19:39:28.451737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.451621    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" event={"ID":"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb","Type":"ContainerDied","Data":"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"}
Oct 30 19:39:28.451737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.451673    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj" event={"ID":"6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb","Type":"ContainerDied","Data":"7bfe7c674bf82ae4ba1266f31b4fc0d5b64b3cc535cbca315023eee71800972a"}
Oct 30 19:39:28.451737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.451707    5085 scope.go:122] "RemoveContainer" containerID="39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"
Oct 30 19:39:28.452003 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.451793    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"
Oct 30 19:39:28.463703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.463679    5085 scope.go:122] "RemoveContainer" containerID="39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"
Oct 30 19:39:28.464061 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:28.464028    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e\": container with ID starting with 39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e not found: ID does not exist" containerID="39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"
Oct 30 19:39:28.464142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.464077    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e"} err="failed to get container status \"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e\": rpc error: code = NotFound desc = could not find container \"39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e\": container with ID starting with 39bbada5d4c374d785972c977cc9e691ca7c3adbb758645f850749422132738e not found: ID does not exist"
Oct 30 19:39:28.599102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.599079    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 19:39:28.600513 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:39:28.600489    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb3c5d_f95c_47bc_990d_5a91e35d3d2f.slice/crio-1a120d14265f006729d7de9ce4f460300274127531e39bacb1fc56d9749c7b79 WatchSource:0}: Error finding container 1a120d14265f006729d7de9ce4f460300274127531e39bacb1fc56d9749c7b79: Status 404 returned error can't find the container with id 1a120d14265f006729d7de9ce4f460300274127531e39bacb1fc56d9749c7b79
Oct 30 19:39:28.602857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.602826    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:28.602954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.602890    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:28.602954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.602927    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:39:28.608903 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.608880    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:39:28.613482 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.613461    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3df7da3e-7f9b7b6747-5hgcj"]
Oct 30 19:39:28.630320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.630294    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5880ef95-c031-4e84-8124-35d8a8c2701d" path="/var/lib/kubelet/pods/5880ef95-c031-4e84-8124-35d8a8c2701d/volumes"
Oct 30 19:39:28.630705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.630692    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" path="/var/lib/kubelet/pods/6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb/volumes"
Oct 30 19:39:28.781723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:28.781700    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"]
Oct 30 19:39:28.783648 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:39:28.783621    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911567bd_a659_47b5_9cb3_b4988388357d.slice/crio-44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018 WatchSource:0}: Error finding container 44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018: Status 404 returned error can't find the container with id 44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018
Oct 30 19:39:29.460877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.460838    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerStarted","Data":"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"}
Oct 30 19:39:29.461366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.460905    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerStarted","Data":"1a120d14265f006729d7de9ce4f460300274127531e39bacb1fc56d9749c7b79"}
Oct 30 19:39:29.463454 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.463427    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs" event={"ID":"911567bd-a659-47b5-9cb3-b4988388357d","Type":"ContainerStarted","Data":"0850f79716161660b46984e95ce326c3875badf29f93b8f3f7c7c66312a62a9d"}
Oct 30 19:39:29.463637 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.463475    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs" event={"ID":"911567bd-a659-47b5-9cb3-b4988388357d","Type":"ContainerStarted","Data":"44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018"}
Oct 30 19:39:29.576304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.576247    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podStartSLOduration=2.5762320279999997 podStartE2EDuration="2.576232028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:29.575287215 +0000 UTC m=+879.079690981" watchObservedRunningTime="2026-10-30 19:39:29.576232028 +0000 UTC m=+879.080635775"
Oct 30 19:39:29.684919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:29.684830    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs" podStartSLOduration=2.684812064 podStartE2EDuration="2.684812064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:29.684305769 +0000 UTC m=+879.188709516" watchObservedRunningTime="2026-10-30 19:39:29.684812064 +0000 UTC m=+879.189215837"
Oct 30 19:39:30.013267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.013232    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:39:30.154033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.153633    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:30.182715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.181915    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:30.471879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.471847    5085 generic.go:356] "Generic (PLEG): container finished" podID="911567bd-a659-47b5-9cb3-b4988388357d" containerID="0850f79716161660b46984e95ce326c3875badf29f93b8f3f7c7c66312a62a9d" exitCode=0
Oct 30 19:39:30.472257 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.471930    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs" event={"ID":"911567bd-a659-47b5-9cb3-b4988388357d","Type":"ContainerDied","Data":"0850f79716161660b46984e95ce326c3875badf29f93b8f3f7c7c66312a62a9d"}
Oct 30 19:39:30.472331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.472315    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:30.473221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:30.473203    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:39:31.622985 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.622959    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:31.732131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.732043    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\" (UniqueName: \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\") pod \"911567bd-a659-47b5-9cb3-b4988388357d\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") "
Oct 30 19:39:31.732305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.732166    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\") pod \"911567bd-a659-47b5-9cb3-b4988388357d\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") "
Oct 30 19:39:31.732305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.732229    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\") pod \"911567bd-a659-47b5-9cb3-b4988388357d\" (UID: \"911567bd-a659-47b5-9cb3-b4988388357d\") "
Oct 30 19:39:31.732652 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.732623    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca" pod "911567bd-a659-47b5-9cb3-b4988388357d" (UID: "911567bd-a659-47b5-9cb3-b4988388357d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:31.734125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.734100    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj" pod "911567bd-a659-47b5-9cb3-b4988388357d" (UID: "911567bd-a659-47b5-9cb3-b4988388357d"). InnerVolumeSpecName "kube-api-access-999hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:31.734219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.734151    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca" pod "911567bd-a659-47b5-9cb3-b4988388357d" (UID: "911567bd-a659-47b5-9cb3-b4988388357d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:31.833707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.833676    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:31.833707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.833703    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/911567bd-a659-47b5-9cb3-b4988388357d-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:31.833707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:31.833714    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999hj\" (UniqueName: \"kubernetes.io/projected/911567bd-a659-47b5-9cb3-b4988388357d-kube-api-access-999hj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:32.487629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:32.487602    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"
Oct 30 19:39:32.487800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:32.487624    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs" event={"ID":"911567bd-a659-47b5-9cb3-b4988388357d","Type":"ContainerDied","Data":"44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018"}
Oct 30 19:39:32.487800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:32.487671    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44dba1826cf93e9f9d8459338b700d7cec65ec3cc871f54c3bb3631327175018"
Oct 30 19:39:43.187597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:43.187547    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:39:43.188098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:43.187828    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy" containerID="cri-o://d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29" gracePeriod=30
Oct 30 19:39:48.464489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.464424    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:48.590855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.590816    5085 generic.go:356] "Generic (PLEG): container finished" podID="c26b2434-6d58-488a-b2af-13b51121fccf" containerID="d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29" exitCode=0
Oct 30 19:39:48.591046 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.590869    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"
Oct 30 19:39:48.591046 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.590897    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerDied","Data":"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"}
Oct 30 19:39:48.591046 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.590958    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262" event={"ID":"c26b2434-6d58-488a-b2af-13b51121fccf","Type":"ContainerDied","Data":"d520701bb1bc4a8404a960ca537d134bb7b02e4c28a7173e4377d0a0aab76259"}
Oct 30 19:39:48.591046 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.590985    5085 scope.go:122] "RemoveContainer" containerID="d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"
Oct 30 19:39:48.600898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.600877    5085 scope.go:122] "RemoveContainer" containerID="37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"
Oct 30 19:39:48.606126 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606097    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606243 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606142    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606243 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606171    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606253    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606327    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606449 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606367    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606449 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606434    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606439    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:48.606576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606497    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606505    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:48.606576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606535    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:48.606770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606594    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\") pod \"c26b2434-6d58-488a-b2af-13b51121fccf\" (UID: \"c26b2434-6d58-488a-b2af-13b51121fccf\") "
Oct 30 19:39:48.606821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.606790    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:48.607329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.607051    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.607329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.607078    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.607329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.607111    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.607329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.607125    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.607329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.607298    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:48.608521 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.608499    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:48.608688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.608660    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:48.608875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.608855    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "kube-api-access-fmxkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:48.608928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.608900    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy" pod "c26b2434-6d58-488a-b2af-13b51121fccf" (UID: "c26b2434-6d58-488a-b2af-13b51121fccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:48.610502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.610483    5085 scope.go:122] "RemoveContainer" containerID="d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"
Oct 30 19:39:48.610832 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:48.610809    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29\": container with ID starting with d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29 not found: ID does not exist" containerID="d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"
Oct 30 19:39:48.610935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.610845    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29"} err="failed to get container status \"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29\": rpc error: code = NotFound desc = could not find container \"d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29\": container with ID starting with d3b33f563546ddbdc16a34f873e2685f726a721d5e9f293038c82d3b0177db29 not found: ID does not exist"
Oct 30 19:39:48.610935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.610869    5085 scope.go:122] "RemoveContainer" containerID="37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"
Oct 30 19:39:48.611137 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:48.611119    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927\": container with ID starting with 37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927 not found: ID does not exist" containerID="37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"
Oct 30 19:39:48.611175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.611143    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927"} err="failed to get container status \"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927\": rpc error: code = NotFound desc = could not find container \"37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927\": container with ID starting with 37f71bfdc3ec17be3a4227b858cb66c26dcf3f2b1a138955386da11395509927 not found: ID does not exist"
Oct 30 19:39:48.707982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.707946    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26b2434-6d58-488a-b2af-13b51121fccf-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.707982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.707981    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26b2434-6d58-488a-b2af-13b51121fccf-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.708270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.707994    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxkd\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-kube-api-access-fmxkd\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.708270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.708008    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26b2434-6d58-488a-b2af-13b51121fccf-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:48.708270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:48.708022    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26b2434-6d58-488a-b2af-13b51121fccf-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:49.001940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:49.001901    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:39:49.006248 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:49.006223    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3df7da3e-openshift-default-5485798879-8x262"]
Oct 30 19:39:50.630766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:50.630726    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" path="/var/lib/kubelet/pods/c26b2434-6d58-488a-b2af-13b51121fccf/volumes"
Oct 30 19:39:50.717683 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:50.717643    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:39:50.724008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:50.723983    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:39:51.938987 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:39:51.938944    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:39:58.606085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:58.606044    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:39:58.945923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:58.945884    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"]
Oct 30 19:39:59.020612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:59.020546    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f214434c-5v7xz"]
Oct 30 19:39:59.034342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:59.033848    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:39:59.034342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:39:59.034129    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" containerID="cri-o://2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22" gracePeriod=30
Oct 30 19:40:00.462589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.458368    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:40:00.478219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478181    5085 cpu_manager.go:414] "containerMap: removing container" podUID="911567bd-a659-47b5-9cb3-b4988388357d" containerName="revoke-keys"
Oct 30 19:40:00.478447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478430    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="911567bd-a659-47b5-9cb3-b4988388357d" containerName="revoke-keys"
Oct 30 19:40:00.478601 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478588    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.478702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478689    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.478791 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478780    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.478868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478858    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.479010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.478996    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" containerName="payload-pre-processing"
Oct 30 19:40:00.479106 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.479093    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" containerName="payload-pre-processing"
Oct 30 19:40:00.479340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.479324    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe4b87c-5f7f-4fcc-ac52-493bd639ccfb" containerName="payload-pre-processing"
Oct 30 19:40:00.479441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.479430    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="911567bd-a659-47b5-9cb3-b4988388357d" containerName="revoke-keys"
Oct 30 19:40:00.479531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.479520    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.479641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.479629    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26b2434-6d58-488a-b2af-13b51121fccf" containerName="istio-proxy"
Oct 30 19:40:00.488336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.488313    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.542788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.542757    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bbfa07-openshift-default-dockercfg-vj7kd\""
Oct 30 19:40:00.593300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.593262    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:40:00.611688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.608500    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"]
Oct 30 19:40:00.613625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.613593    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-be15b010-vc9zs"]
Oct 30 19:40:00.631448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.631410    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="911567bd-a659-47b5-9cb3-b4988388357d" path="/var/lib/kubelet/pods/911567bd-a659-47b5-9cb3-b4988388357d/volumes"
Oct 30 19:40:00.632014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.631993    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9869b5f1-928c-4765-b944-dd85f0f82d76" path="/var/lib/kubelet/pods/9869b5f1-928c-4765-b944-dd85f0f82d76/volumes"
Oct 30 19:40:00.633696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633664    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633706    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633768    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633859    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9pv\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633895    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633938    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.633988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.633969    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.634196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.634120    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.634196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.634156    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.690715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.690683    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:40:00.709025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.708979    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy" containerID="cri-o://e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd" gracePeriod=30
Oct 30 19:40:00.712474 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.712442    5085 patch_prober.go:29] interesting pod/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:40:00.712716 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.712548    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:40:00.735204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735172    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9pv\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735211    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735246    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735289    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735328    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735359    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735453    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735478    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735506    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735787    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735888 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735867    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.735947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.735925    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.736045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.736023    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.736206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.736185    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.737792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.737775    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.737937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.737922    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.799734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.799550    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.799922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.799896    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9pv\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:00.822993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:00.822966    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:01.285631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.285596    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:40:01.287870 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:40:01.287831    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e93fa0_525e_47a6_97a9_82089a0a7334.slice/crio-065a3707fedc30f3cdd32cdc953c3be7d6fe0cca310a4108e6fec87a268be7b1 WatchSource:0}: Error finding container 065a3707fedc30f3cdd32cdc953c3be7d6fe0cca310a4108e6fec87a268be7b1: Status 404 returned error can't find the container with id 065a3707fedc30f3cdd32cdc953c3be7d6fe0cca310a4108e6fec87a268be7b1
Oct 30 19:40:01.290330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.290293    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:01.290410 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.290378    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:01.290451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.290421    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:01.724892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.724855    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerStarted","Data":"64bb35533ab7d80e7f22fca5cadcf3d62b8e9edf685b3c44b7c20a1acad85ad6"}
Oct 30 19:40:01.725268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.724905    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerStarted","Data":"065a3707fedc30f3cdd32cdc953c3be7d6fe0cca310a4108e6fec87a268be7b1"}
Oct 30 19:40:01.823697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.823660    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:01.847241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:01.847192    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podStartSLOduration=1.847177305 podStartE2EDuration="1.847177305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:01.84450198 +0000 UTC m=+911.348905727" watchObservedRunningTime="2026-10-30 19:40:01.847177305 +0000 UTC m=+911.351581111"
Oct 30 19:40:02.088241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.088150    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 19:40:02.091925 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.091903    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.147918 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.144948    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-98a19b-openshift-default-dockercfg-rhphv\""
Oct 30 19:40:02.179594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.178964    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 19:40:02.190231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190190    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190268    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190299    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190330    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190362    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190460    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190501    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190535    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.190723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.190613    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrgp\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292086 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292053    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292268    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292396    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrgp\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292510    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292620    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292726    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292858    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.292933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292880    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293034 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.292895    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.293357    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293464 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.293407    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.293576    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.293622    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.293769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.293748    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.295239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.295218    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.295922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.295901    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.350815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.350746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.351018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.350995    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrgp\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.406023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.405989    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:02.828883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.828848    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:02.852184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.852155    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 19:40:02.853180 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:40:02.853148    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea11f0c_7bc5_4c90_8882_df523acac165.slice/crio-8e22d8f8664a69c0331d723291f01d0ac2ab448d01da3c1fb59694bf4a926071 WatchSource:0}: Error finding container 8e22d8f8664a69c0331d723291f01d0ac2ab448d01da3c1fb59694bf4a926071: Status 404 returned error can't find the container with id 8e22d8f8664a69c0331d723291f01d0ac2ab448d01da3c1fb59694bf4a926071
Oct 30 19:40:02.855718 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.855686    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:02.855799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.855759    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:02.855799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:02.855790    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:03.738995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:03.738955    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerStarted","Data":"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"}
Oct 30 19:40:03.739200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:03.739016    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerStarted","Data":"8e22d8f8664a69c0331d723291f01d0ac2ab448d01da3c1fb59694bf4a926071"}
Oct 30 19:40:03.739298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:03.739283    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:03.740356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:03.740335    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:03.902057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:03.901998    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podStartSLOduration=1.901982049 podStartE2EDuration="1.901982049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:03.90036093 +0000 UTC m=+913.404764676" watchObservedRunningTime="2026-10-30 19:40:03.901982049 +0000 UTC m=+913.406385796"
Oct 30 19:40:04.255024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.254993    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:40:04.406948 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.406851    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:04.412866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.412841    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:04.419528 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419501    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419573    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419691    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419790    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419835    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419865    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419904    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.419997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419959    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:04.419997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.419977    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.420161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420005    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\") pod \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\" (UID: \"37c3d63a-b9dc-4726-aeb9-8adb92c7695c\") "
Oct 30 19:40:04.420161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420006    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:04.420285 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420264    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:04.420445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420401    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:04.420631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420536    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:04.420895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420753    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.420895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420778    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.420895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420792    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.420895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420805    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.420895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.420818    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.422519 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.422494    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:04.422612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.422499    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:04.422932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.422906    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "kube-api-access-d9xdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:04.423093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.423077    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token" pod "37c3d63a-b9dc-4726-aeb9-8adb92c7695c" (UID: "37c3d63a-b9dc-4726-aeb9-8adb92c7695c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:04.522389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.522346    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xdr\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-kube-api-access-d9xdr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.522389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.522388    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.522389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.522400    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.522654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.522409    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37c3d63a-b9dc-4726-aeb9-8adb92c7695c-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:04.747004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.746969    5085 generic.go:356] "Generic (PLEG): container finished" podID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerID="2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22" exitCode=0
Oct 30 19:40:04.747210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.747044    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"
Oct 30 19:40:04.747210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.747047    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerDied","Data":"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"}
Oct 30 19:40:04.747210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.747125    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7" event={"ID":"37c3d63a-b9dc-4726-aeb9-8adb92c7695c","Type":"ContainerDied","Data":"9a61a1e68f1c36a545c99d71c1c0eee154b06960d3afe274cafd854f5ff965d3"}
Oct 30 19:40:04.747210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.747164    5085 scope.go:122] "RemoveContainer" containerID="2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"
Oct 30 19:40:04.749055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.748915    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:04.751206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.751184    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:04.764509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.764487    5085 scope.go:122] "RemoveContainer" containerID="9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"
Oct 30 19:40:04.778576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.778541    5085 scope.go:122] "RemoveContainer" containerID="2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"
Oct 30 19:40:04.779004 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:04.778977    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22\": container with ID starting with 2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22 not found: ID does not exist" containerID="2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"
Oct 30 19:40:04.779122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.779016    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22"} err="failed to get container status \"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22\": rpc error: code = NotFound desc = could not find container \"2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22\": container with ID starting with 2d2fa357a0d49b4fd945d6815193fdec38992124c984f7c888ab0796d04b3b22 not found: ID does not exist"
Oct 30 19:40:04.779122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.779043    5085 scope.go:122] "RemoveContainer" containerID="9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"
Oct 30 19:40:04.779455 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:04.779420    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908\": container with ID starting with 9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908 not found: ID does not exist" containerID="9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"
Oct 30 19:40:04.779508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:04.779457    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908"} err="failed to get container status \"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908\": rpc error: code = NotFound desc = could not find container \"9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908\": container with ID starting with 9f8c032e021c0efad8a1d0821eceaca7dd443993f5ec573b8de532e4a2b80908 not found: ID does not exist"
Oct 30 19:40:05.020857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:05.020775    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:40:05.025630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:05.025598    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f214434c-gw-openshift-default-58f59df989-pvzs7"]
Oct 30 19:40:05.985915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:05.985841    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:40:06.034357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034317    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034358    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034390    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034429    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034460    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034596    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034664    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.034955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034711    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.035273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.034973    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:06.035273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035016    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:06.035381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035321    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:06.035381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035365    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:40:06.035536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035514    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\") pod \"4212db21-597f-4e1c-a535-53eeadc3c324\" (UID: \"4212db21-597f-4e1c-a535-53eeadc3c324\") "
Oct 30 19:40:06.035808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035788    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:06.035960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035945    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.035997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035966    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4212db21-597f-4e1c-a535-53eeadc3c324-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.035997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035981    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.036060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.035995    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.036060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.036008    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.037205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.037181    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:40:06.037310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.037200    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "kube-api-access-5hqlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:06.037310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.037267    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:40:06.037612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.037597    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy" pod "4212db21-597f-4e1c-a535-53eeadc3c324" (UID: "4212db21-597f-4e1c-a535-53eeadc3c324"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:40:06.137094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.137056    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4212db21-597f-4e1c-a535-53eeadc3c324-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.137094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.137089    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqlt\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-kube-api-access-5hqlt\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.137094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.137103    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4212db21-597f-4e1c-a535-53eeadc3c324-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.137374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.137117    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4212db21-597f-4e1c-a535-53eeadc3c324-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:40:06.632434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.632388    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" path="/var/lib/kubelet/pods/37c3d63a-b9dc-4726-aeb9-8adb92c7695c/volumes"
Oct 30 19:40:06.763473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.763439    5085 generic.go:356] "Generic (PLEG): container finished" podID="4212db21-597f-4e1c-a535-53eeadc3c324" containerID="e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd" exitCode=0
Oct 30 19:40:06.763691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.763493    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"
Oct 30 19:40:06.763691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.763526    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerDied","Data":"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"}
Oct 30 19:40:06.763691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.763609    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt" event={"ID":"4212db21-597f-4e1c-a535-53eeadc3c324","Type":"ContainerDied","Data":"fabf182236a73258a4a24b0a225ac50df490a1640abc2fb4582e1ddf5bdb33ed"}
Oct 30 19:40:06.763691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.763638    5085 scope.go:122] "RemoveContainer" containerID="e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"
Oct 30 19:40:06.773731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.773708    5085 scope.go:122] "RemoveContainer" containerID="2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"
Oct 30 19:40:06.785031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.785006    5085 scope.go:122] "RemoveContainer" containerID="e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"
Oct 30 19:40:06.785366 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:06.785341    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd\": container with ID starting with e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd not found: ID does not exist" containerID="e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"
Oct 30 19:40:06.785438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.785380    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd"} err="failed to get container status \"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd\": rpc error: code = NotFound desc = could not find container \"e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd\": container with ID starting with e814cbdb86cf6c86e93923e22806860df960429fde7741952bcec070ed559ffd not found: ID does not exist"
Oct 30 19:40:06.785438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.785406    5085 scope.go:122] "RemoveContainer" containerID="2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"
Oct 30 19:40:06.785691 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:06.785663    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6\": container with ID starting with 2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6 not found: ID does not exist" containerID="2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"
Oct 30 19:40:06.785761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.785700    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6"} err="failed to get container status \"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6\": rpc error: code = NotFound desc = could not find container \"2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6\": container with ID starting with 2af72729e485f2d316cf5a013f292724364180cf79211d80d7912cf0c04908f6 not found: ID does not exist"
Oct 30 19:40:06.967207 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:06.967171    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:40:07.014108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:07.014062    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-be15b010-openshift-default-78cc9bcbbf-mbmxt"]
Oct 30 19:40:08.630501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.630465    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" path="/var/lib/kubelet/pods/4212db21-597f-4e1c-a535-53eeadc3c324/volumes"
Oct 30 19:40:08.778953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.778904    5085 generic.go:356] "Generic (PLEG): container finished" podID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerID="3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b" exitCode=0
Oct 30 19:40:08.779160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.778990    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerDied","Data":"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"}
Oct 30 19:40:08.779692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.779670    5085 scope.go:122] "RemoveContainer" containerID="3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"
Oct 30 19:40:08.780971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.780940    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:08.781073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.781020    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:08.781073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:08.781066    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:09.804182 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:09.804141    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerStarted","Data":"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"}
Oct 30 19:40:10.030551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:10.030513    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:40:10.150419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:10.150309    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:40:10.150637 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:10.150616    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:40:10.150703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:10.150663    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 19:40:11.150795 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:11.150757    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:40:11.151210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:11.150809    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 19:40:12.150735 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:12.150696    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:40:12.150925 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:12.150745    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 19:40:13.149912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:13.149872    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:40:13.150103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:13.149940    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 19:40:14.150634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:14.150595    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:40:14.151042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:14.150656    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused"
Oct 30 19:40:15.399424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:15.399386    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:40:15.400075 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:15.400042    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:40:15.400934 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:15.400912    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 19:40:21.939469 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:21.939432    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:40:22.600798 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.600756    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 19:40:22.601307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601288    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.601405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601307    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.601405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601333    5085 cpu_manager.go:414] "containerMap: removing container" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:22.601405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601341    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:22.601405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601369    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.601405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601377    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.601682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601505    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.601682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601521    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:22.601682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.601534    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4212db21-597f-4e1c-a535-53eeadc3c324" containerName="istio-proxy"
Oct 30 19:40:22.604723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.604702    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.656501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.656468    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"openshift-service-ca.crt\""
Oct 30 19:40:22.656714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.656662    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"kube-root-ca.crt\""
Oct 30 19:40:22.703495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.703455    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"default-dockercfg-4fxps\""
Oct 30 19:40:22.703705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.703547    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a74d15\"/\"simulator-w0-a74d15-kserve-self-signed-certs\""
Oct 30 19:40:22.706391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.706371    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 19:40:22.730237 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730208    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.730368 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730244    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.730408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730363    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.730408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730398    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9bw\" (UniqueName: \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.730503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730485    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.730540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.730502    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.831789 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.831750    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.831988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.831799    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.831988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.831868    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.831988 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.831894    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9bw\" (UniqueName: \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.832171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.831991    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.832171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.832017    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.832287 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.832197    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.832357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.832313    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.832460 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.832364    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.834331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.834306    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.834465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.834449    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.891299 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.891235    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9bw\" (UniqueName: \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\") pod \"simulator-w0-a74d15-kserve-7fd84895c7-rbtrc\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") " pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:22.916068 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:22.916037    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:23.287671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:23.287646    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 19:40:23.289217 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:40:23.289189    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod291fdcd5_7ff1_4eca_be25_dfb1423d126b.slice/crio-123157c30a0a78b6e899379f8cd2fb6f3615af35223a278587a55a908c6962a7 WatchSource:0}: Error finding container 123157c30a0a78b6e899379f8cd2fb6f3615af35223a278587a55a908c6962a7: Status 404 returned error can't find the container with id 123157c30a0a78b6e899379f8cd2fb6f3615af35223a278587a55a908c6962a7
Oct 30 19:40:23.886056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:23.886017    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" event={"ID":"291fdcd5-7ff1-4eca-be25-dfb1423d126b","Type":"ContainerStarted","Data":"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"}
Oct 30 19:40:23.886240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:23.886063    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" event={"ID":"291fdcd5-7ff1-4eca-be25-dfb1423d126b","Type":"ContainerStarted","Data":"123157c30a0a78b6e899379f8cd2fb6f3615af35223a278587a55a908c6962a7"}
Oct 30 19:40:23.916511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:23.916480    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:23.998380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:23.998320    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podStartSLOduration=1.998305989 podStartE2EDuration="1.998305989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:23.997085502 +0000 UTC m=+933.501489248" watchObservedRunningTime="2026-10-30 19:40:23.998305989 +0000 UTC m=+933.502709773"
Oct 30 19:40:26.410095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:26.410055    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:40:32.916692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:32.916603    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:32.929109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:32.929086    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:32.957132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:32.957095    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:40:43.912725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:43.912684    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:40:45.937290 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.937256    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:40:45.937749 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.937735    5085 cpu_manager.go:414] "containerMap: removing container" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:45.937794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.937749    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:45.937852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.937843    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c3d63a-b9dc-4726-aeb9-8adb92c7695c" containerName="istio-proxy"
Oct 30 19:40:45.940066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.940045    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988499    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988541    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988580    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988686    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988728    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.988824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988814    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.989056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988881    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.989056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.988913    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkd2c\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:45.989056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:45.989020    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.046820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.046789    5085 generic.go:356] "Generic (PLEG): container finished" podID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerID="64bb35533ab7d80e7f22fca5cadcf3d62b8e9edf685b3c44b7c20a1acad85ad6" exitCode=0
Oct 30 19:40:46.046996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.046867    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerDied","Data":"64bb35533ab7d80e7f22fca5cadcf3d62b8e9edf685b3c44b7c20a1acad85ad6"}
Oct 30 19:40:46.047289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.047274    5085 scope.go:122] "RemoveContainer" containerID="64bb35533ab7d80e7f22fca5cadcf3d62b8e9edf685b3c44b7c20a1acad85ad6"
Oct 30 19:40:46.053279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.053242    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:46.053403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.053319    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:46.053403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.053353    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:46.089696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.089663    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.089911    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkd2c\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090192    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090335    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090545    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090614    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090648    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090692    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.090746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090726    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.091010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090790    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.091010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.090877    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.091136 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.091112    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.091198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.091148    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.091320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.091294    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.092626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.092607    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.093294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.093274    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.147194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.147154    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:40:46.151409 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.151381    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkd2c\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.186593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.180973    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.253399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.253366    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:46.288881 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.288843    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 19:40:46.293778 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.293750    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.345047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.344694    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"openshift-service-ca.crt\""
Oct 30 19:40:46.345047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.344840    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"default-dockercfg-krcrx\""
Oct 30 19:40:46.345047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.344932    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"simulator-w4-4ec609-kserve-self-signed-certs\""
Oct 30 19:40:46.345047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.345012    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ec609\"/\"kube-root-ca.crt\""
Oct 30 19:40:46.346630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.346387    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 19:40:46.394227 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394156    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl249\" (UniqueName: \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.394370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394275    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.394370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394342    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.394501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394375    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.394501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394405    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.394501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.394449    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495573 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495462    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl249\" (UniqueName: \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495550    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495672    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495706    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495732    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.495996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.495773    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.496435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.496319    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.496435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.496386    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.496657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.496637    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.498316 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.498296    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.498463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.498448    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.554164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.554135    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl249\" (UniqueName: \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\") pod \"simulator-w4-4ec609-kserve-f955958fb-9pggh\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") " pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.610482 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.610446    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:46.753644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.751543    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:40:46.755672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.754809    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:46.755672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.754883    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:46.755672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:46.754926    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:47.056521 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.056498    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 19:40:47.072640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.068857    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" event={"ID":"ebaa0b35-1e80-44e0-a3cc-b77492e53a37","Type":"ContainerStarted","Data":"53c9c163ab0a56a4ca7cc62616d1a554284682d0a36838dd7c9009d632e86bf3"}
Oct 30 19:40:47.072640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.068929    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" event={"ID":"ebaa0b35-1e80-44e0-a3cc-b77492e53a37","Type":"ContainerStarted","Data":"c8263eb49d3d20eb923b99596a942ae991f33e42d4dc8f54cece42e5c6e179b7"}
Oct 30 19:40:47.113964 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.113908    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerStarted","Data":"f5c1e3493e24e44b89df656a963a0b3217eea0075119dfc767da5a1468e81c9b"}
Oct 30 19:40:47.254388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.254354    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:47.255623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.255598    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:47.255704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.255646    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:47.317209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.317071    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podStartSLOduration=2.317050207 podStartE2EDuration="2.317050207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.19251348 +0000 UTC m=+956.696917228" watchObservedRunningTime="2026-10-30 19:40:47.317050207 +0000 UTC m=+956.821453954"
Oct 30 19:40:47.830590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.826498    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:47.830787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.830723    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:47.830884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:47.830782    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 19:40:48.141757 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.141659    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" event={"ID":"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545","Type":"ContainerStarted","Data":"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"}
Oct 30 19:40:48.142163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.141803    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" event={"ID":"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545","Type":"ContainerStarted","Data":"bb081b3cd09d70dacae30753be8d926b3f2177d759a1ad6540003e71d9f16c42"}
Oct 30 19:40:48.254184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.254148    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:48.254334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.254197    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:48.269589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.262470    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podStartSLOduration=2.262450471 podStartE2EDuration="2.262450471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:48.260946327 +0000 UTC m=+957.765350087" watchObservedRunningTime="2026-10-30 19:40:48.262450471 +0000 UTC m=+957.766854238"
Oct 30 19:40:48.617589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.616667    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:48.739741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.739709    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:48.830138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.830085    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:48.830350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:48.830152    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 19:40:49.254728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.254694    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:49.255146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.254760    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:49.614287 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.614186    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 19:40:49.634853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.632600    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680211    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcxs\" (UniqueName: \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680288    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680369    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680471    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680505    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.680588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.680594    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.732979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.732948    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"openshift-service-ca.crt\""
Oct 30 19:40:49.736648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.733345    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"simulator-w2-b0a183-kserve-self-signed-certs\""
Oct 30 19:40:49.736912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.733418    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"default-dockercfg-xjrrw\""
Oct 30 19:40:49.737016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.733472    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"kube-root-ca.crt\""
Oct 30 19:40:49.738712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.738689    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 19:40:49.798391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.798339    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.798722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.798690    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.812806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.812773    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.813103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.813062    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcxs\" (UniqueName: \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.816605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.813265    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.832878 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.832832    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.833176 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.821103    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.833324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.822283    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.833512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.823505    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:49.833690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.833659    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 19:40:49.843603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.821351    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.843603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.842982    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.844173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.844144    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.892635 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.892502    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcxs\" (UniqueName: \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\") pod \"simulator-w2-b0a183-kserve-677b4cc58-gxkb9\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") " pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:49.947914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:49.947861    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:50.254734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:50.254679    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:50.254901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:50.254743    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:50.512412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:50.512373    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 19:40:50.823461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:50.823425    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:50.823619 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:50.823484    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 19:40:51.211780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.211739    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" event={"ID":"69ce92c4-2611-41d2-91da-1f76052c76d6","Type":"ContainerStarted","Data":"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"}
Oct 30 19:40:51.211780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.211786    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" event={"ID":"69ce92c4-2611-41d2-91da-1f76052c76d6","Type":"ContainerStarted","Data":"9c4af9dcb87c750e82d514d359557af61ec8e030002326d112c8eca05074b293"}
Oct 30 19:40:51.254366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.254326    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:51.254524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.254393    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:51.336878 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.336368    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podStartSLOduration=2.336354007 podStartE2EDuration="2.336354007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:51.335109291 +0000 UTC m=+960.839513038" watchObservedRunningTime="2026-10-30 19:40:51.336354007 +0000 UTC m=+960.840757792"
Oct 30 19:40:51.824271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.824240    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 30 19:40:51.824486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.824465    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 30 19:40:51.942627 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:40:51.942593    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:40:51.948020 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:51.947918    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:52.254091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:52.254054    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 30 19:40:52.254312 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:52.254116    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 30 19:40:52.828538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:52.828506    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:52.829609 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:52.829584    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:40:53.258935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:53.258902    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:53.259302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:53.259227    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:53.260206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:53.260180    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:40:55.241552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.241516    5085 generic.go:356] "Generic (PLEG): container finished" podID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerID="df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa" exitCode=0
Oct 30 19:40:55.242000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.241591    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerDied","Data":"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"}
Oct 30 19:40:55.242050 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.242008    5085 scope.go:122] "RemoveContainer" containerID="df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"
Oct 30 19:40:55.243498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.243467    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:55.243593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.243557    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:55.243643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.243628    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:40:55.401596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:55.397084    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:40:56.252464 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.252424    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerStarted","Data":"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"}
Oct 30 19:40:56.406720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.406688    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:40:56.406907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.406837    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:56.406907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.406890    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:40:56.610902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.610808    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:56.630198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:56.630169    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:57.298128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:57.298089    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:40:57.406932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:57.406881    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:57.407141 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:57.406946    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:40:58.406521 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:58.406475    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:58.407069 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:58.406534    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:40:59.406889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:59.406851    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:40:59.406889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:59.406906    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:40:59.948205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:59.948173    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:40:59.986079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:40:59.986047    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:41:00.311013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:00.310942    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:41:00.406377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:00.406346    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:41:00.406586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:00.406394    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:41:01.406830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:01.406795    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:41:01.407259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:01.406864    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:41:02.411136 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:02.411105    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:41:02.411534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:02.411398    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:41:02.412035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:02.412017    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 19:41:16.187059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.187019    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:41:16.191201 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.191178    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241080    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241135    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241211    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxjm\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241234    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241264    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241318    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241424    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241544    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.241743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.241685    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.342808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.342763    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.342854    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.342891    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.342929    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343192 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343114    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxjm\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343254 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343189    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343254 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343192    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343254 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343238    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343254    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343282    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343332    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343654    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343703    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.343976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.343955    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.345266 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.345245    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.345723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.345700    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.393962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.393924    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:41:16.403600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.403525    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.403748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.403733    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxjm\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.504581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.504488    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:16.929035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.929007    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:41:16.930733 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:41:16.930704    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a516a0a_7285_451c_be60_9f16f850c43c.slice/crio-25b92534c90ffac2353ed534fae7f97ed4e8bddb85c7a7cf8074546c3ee0ec9e WatchSource:0}: Error finding container 25b92534c90ffac2353ed534fae7f97ed4e8bddb85c7a7cf8074546c3ee0ec9e: Status 404 returned error can't find the container with id 25b92534c90ffac2353ed534fae7f97ed4e8bddb85c7a7cf8074546c3ee0ec9e
Oct 30 19:41:16.933129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.933098    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:16.933199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.933162    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:16.933199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:16.933189    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:17.408509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.408412    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" event={"ID":"6a516a0a-7285-451c-be60-9f16f850c43c","Type":"ContainerStarted","Data":"fd578ca624a089b7bffeca07adff74b8ed6a4c7a4a2338436a73857a5d7f49b7"}
Oct 30 19:41:17.408509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.408474    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" event={"ID":"6a516a0a-7285-451c-be60-9f16f850c43c","Type":"ContainerStarted","Data":"25b92534c90ffac2353ed534fae7f97ed4e8bddb85c7a7cf8074546c3ee0ec9e"}
Oct 30 19:41:17.504840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.504804    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:17.506354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.506330    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:17.506412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.506382    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:17.625502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:17.625445    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podStartSLOduration=1.625430906 podStartE2EDuration="1.625430906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:17.585622664 +0000 UTC m=+987.090026411" watchObservedRunningTime="2026-10-30 19:41:17.625430906 +0000 UTC m=+987.129834685"
Oct 30 19:41:18.505187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:18.505146    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:18.505634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:18.505207    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:19.505686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:19.505654    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:19.506090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:19.505715    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:19.913403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:19.913322    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:41:20.505757 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:20.505724    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:20.506270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:20.505789    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:20.920502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:20.920474    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 19:41:20.925791 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:20.925770    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:20.989901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:20.989873    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 19:41:21.018408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018373    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.018582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018422    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.018582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018501    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qbz6\" (UniqueName: \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.018582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018544    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.018696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018634    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.018812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.018787    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.033891 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.033861    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"openshift-service-ca.crt\""
Oct 30 19:41:21.034041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.033912    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"kube-root-ca.crt\""
Oct 30 19:41:21.034041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.033949    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"default-dockercfg-r6p6l\""
Oct 30 19:41:21.034041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.033969    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bbfa07\"/\"simulator-w3-bbfa07-kserve-self-signed-certs\""
Oct 30 19:41:21.119655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119621    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.119655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119658    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.119909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119699    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qbz6\" (UniqueName: \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.119909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119734    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.119909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119790    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.120045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.119907    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.120096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.120074    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.120497 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.120469    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.120789 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.120766    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.122088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.122068    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.122458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.122440    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.184701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.184622    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qbz6\" (UniqueName: \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\") pod \"simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") " pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.238036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.237999    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:21.505603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.505506    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:21.505741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.505594    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:21.644926 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:21.644902    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 19:41:21.646333 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:41:21.646309    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8fc380b_ae22_4c61_a47e_5220c0baef93.slice/crio-0ab5581e46dae76c4355d43d6970dfbc1a12c80e5b1cc95b4e94d2daf6579ff7 WatchSource:0}: Error finding container 0ab5581e46dae76c4355d43d6970dfbc1a12c80e5b1cc95b4e94d2daf6579ff7: Status 404 returned error can't find the container with id 0ab5581e46dae76c4355d43d6970dfbc1a12c80e5b1cc95b4e94d2daf6579ff7
Oct 30 19:41:21.939600 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:41:21.939552    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:41:22.442552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:22.442515    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" event={"ID":"c8fc380b-ae22-4c61-a47e-5220c0baef93","Type":"ContainerStarted","Data":"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"}
Oct 30 19:41:22.442552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:22.442557    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" event={"ID":"c8fc380b-ae22-4c61-a47e-5220c0baef93","Type":"ContainerStarted","Data":"0ab5581e46dae76c4355d43d6970dfbc1a12c80e5b1cc95b4e94d2daf6579ff7"}
Oct 30 19:41:22.505099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:22.505064    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:41:22.505310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:22.505115    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 30 19:41:23.238230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:23.238194    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:23.508876 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:23.508796    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:23.509240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:23.509220    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:23.509805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:23.509785    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:41:23.639953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:23.639901    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podStartSLOduration=3.639887397 podStartE2EDuration="3.639887397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:22.58179658 +0000 UTC m=+992.086200328" watchObservedRunningTime="2026-10-30 19:41:23.639887397 +0000 UTC m=+993.144291167"
Oct 30 19:41:31.239107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.239070    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:31.252132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.252103    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:31.512845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.512771    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:41:31.933504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.933470    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-l5qjs-nmczz"]
Oct 30 19:41:31.938404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.938385    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:31.995083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:31.995050    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l5qjs-nmczz"]
Oct 30 19:41:32.055645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.055609    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.055853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.055656    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tqvj\" (UniqueName: \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.055853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.055829    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.055959 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.055870    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157128    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157185    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tqvj\" (UniqueName: \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157296    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157326    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157783    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.157984 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.157965    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.158089 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.158070    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.215954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.215887    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tqvj\" (UniqueName: \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\") pod \"periodic-gathering-l5qjs-nmczz\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") " pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.251719 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.251693    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:41:32.631765 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:41:32.631731    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34d47fb_addd_4b47_8692_d378137ffac1.slice/crio-daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4 WatchSource:0}: Error finding container daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4: Status 404 returned error can't find the container with id daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4
Oct 30 19:41:32.632326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:32.632305    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l5qjs-nmczz"]
Oct 30 19:41:33.522951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:33.522909    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5qjs-nmczz" event={"ID":"f34d47fb-addd-4b47-8692-d378137ffac1","Type":"ContainerStarted","Data":"56861f3ebdc7885a2d018aeef4fccb722a7c64a861aba3ed8d2cb8323d29f88e"}
Oct 30 19:41:33.523430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:33.522961    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5qjs-nmczz" event={"ID":"f34d47fb-addd-4b47-8692-d378137ffac1","Type":"ContainerStarted","Data":"daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4"}
Oct 30 19:41:33.633349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:33.633288    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-l5qjs-nmczz" podStartSLOduration=2.633271285 podStartE2EDuration="2.633271285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:33.632712508 +0000 UTC m=+1003.137116255" watchObservedRunningTime="2026-10-30 19:41:33.633271285 +0000 UTC m=+1003.137675034"
Oct 30 19:41:35.616436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:35.616403    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:41:44.292332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.292299    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 19:41:44.296493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.296472    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.346324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.346296    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a74d15-openshift-default-dockercfg-pfbt9\""
Oct 30 19:41:44.389417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389380    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389437    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389607    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389640    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389674    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389741    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krgxg\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389788    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389803    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.389898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.389832    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.400484 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.400457    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 19:41:44.491245 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491216    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491422 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491276    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491422 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491326    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491419    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491465    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491496    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491665    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krgxg\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491699    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491731    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491777    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.491895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491802    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.492002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.491979    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.492057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.492000    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.492343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.492311    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.493812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.493785    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.493904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.493872    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.552115 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.552039    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krgxg\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.552295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.552271    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:44.609963 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:44.609931    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:45.064320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.064289    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 19:41:45.065425 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:41:45.065396    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75cb793_10a5_431b_9086_130755cb8381.slice/crio-3839bbda6208b56f71bd076d12811eaeddf272ab00262c3f709a0668f4413e52 WatchSource:0}: Error finding container 3839bbda6208b56f71bd076d12811eaeddf272ab00262c3f709a0668f4413e52: Status 404 returned error can't find the container with id 3839bbda6208b56f71bd076d12811eaeddf272ab00262c3f709a0668f4413e52
Oct 30 19:41:45.067972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.067938    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:45.068042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.068006    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:45.068042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.068031    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:41:45.598643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.598608    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" event={"ID":"c75cb793-10a5-431b-9086-130755cb8381","Type":"ContainerStarted","Data":"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"}
Oct 30 19:41:45.599047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.598676    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" event={"ID":"c75cb793-10a5-431b-9086-130755cb8381","Type":"ContainerStarted","Data":"3839bbda6208b56f71bd076d12811eaeddf272ab00262c3f709a0668f4413e52"}
Oct 30 19:41:45.610142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.610116    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:45.778586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:45.778001    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podStartSLOduration=1.777979272 podStartE2EDuration="1.777979272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:45.761004341 +0000 UTC m=+1015.265408128" watchObservedRunningTime="2026-10-30 19:41:45.777979272 +0000 UTC m=+1015.282383020"
Oct 30 19:41:46.612628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:46.612591    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:41:46.613040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:46.612643    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:41:47.611011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:47.610972    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:41:47.611219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:47.611059    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:41:48.611262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:48.611225    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:41:48.611760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:48.611286    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:41:49.610723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:49.610688    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:41:49.610898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:49.610755    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:41:50.610462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:50.610424    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:41:50.610848 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:50.610492    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:41:50.934732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:50.934698    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:41:51.950584 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:41:51.947317    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:41:52.652818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:52.652782    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:41:52.653023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:52.652845    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:52.698481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:52.698445    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:52.699892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:52.699870    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:41:52.700721 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:41:52.700700    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 19:42:17.256857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:17.256822    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:42:21.940123 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:42:21.940085    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:42:29.568458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.568423    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"]
Oct 30 19:42:29.574010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.573985    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.627060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.627022    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"]
Oct 30 19:42:29.670400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.670367    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:29.670400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.670389    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:42:29.694391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.694354    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.694586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.694523    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6km\" (UniqueName: \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.694666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.694596    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.795275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.795232    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6km\" (UniqueName: \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.795275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.795279    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.795522 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.795379    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.795971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.795935    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.797788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.797768    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.854259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.854183    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6km\" (UniqueName: \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\") pod \"maas-api-revoke-sub-6bfeb379-ns879\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:29.887021 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:29.886991    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:30.257521 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:30.257490    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"]
Oct 30 19:42:30.263069 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:42:30.263036    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0e0a33_294b_4a98_baeb_65d299391c17.slice/crio-fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5 WatchSource:0}: Error finding container fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5: Status 404 returned error can't find the container with id fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5
Oct 30 19:42:30.948790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:30.948754    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879" event={"ID":"7a0e0a33-294b-4a98-baeb-65d299391c17","Type":"ContainerStarted","Data":"2a3892ea923fa0ad2d3205a59f962a646f4803ac0716016338dc28b0343467d5"}
Oct 30 19:42:30.949276 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:30.948815    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879" event={"ID":"7a0e0a33-294b-4a98-baeb-65d299391c17","Type":"ContainerStarted","Data":"fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5"}
Oct 30 19:42:31.061445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:31.061388    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879" podStartSLOduration=2.061372888 podStartE2EDuration="2.061372888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:31.059539472 +0000 UTC m=+1060.563943218" watchObservedRunningTime="2026-10-30 19:42:31.061372888 +0000 UTC m=+1060.565776634"
Oct 30 19:42:31.957649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:31.957616    5085 generic.go:356] "Generic (PLEG): container finished" podID="7a0e0a33-294b-4a98-baeb-65d299391c17" containerID="2a3892ea923fa0ad2d3205a59f962a646f4803ac0716016338dc28b0343467d5" exitCode=0
Oct 30 19:42:31.958030 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:31.957699    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879" event={"ID":"7a0e0a33-294b-4a98-baeb-65d299391c17","Type":"ContainerDied","Data":"2a3892ea923fa0ad2d3205a59f962a646f4803ac0716016338dc28b0343467d5"}
Oct 30 19:42:33.567099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.566752    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:33.628001 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.627967    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\" (UniqueName: \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\") pod \"7a0e0a33-294b-4a98-baeb-65d299391c17\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") "
Oct 30 19:42:33.628804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.628779    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\") pod \"7a0e0a33-294b-4a98-baeb-65d299391c17\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") "
Oct 30 19:42:33.631840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.631819    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\") pod \"7a0e0a33-294b-4a98-baeb-65d299391c17\" (UID: \"7a0e0a33-294b-4a98-baeb-65d299391c17\") "
Oct 30 19:42:33.634860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.631756    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca" pod "7a0e0a33-294b-4a98-baeb-65d299391c17" (UID: "7a0e0a33-294b-4a98-baeb-65d299391c17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:33.641662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.641546    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca" pod "7a0e0a33-294b-4a98-baeb-65d299391c17" (UID: "7a0e0a33-294b-4a98-baeb-65d299391c17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:33.646662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.646629    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km" pod "7a0e0a33-294b-4a98-baeb-65d299391c17" (UID: "7a0e0a33-294b-4a98-baeb-65d299391c17"). InnerVolumeSpecName "kube-api-access-sm6km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:33.735920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.735667    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6km\" (UniqueName: \"kubernetes.io/projected/7a0e0a33-294b-4a98-baeb-65d299391c17-kube-api-access-sm6km\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:33.735920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.735705    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:33.735920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.735719    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a0e0a33-294b-4a98-baeb-65d299391c17-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:33.977448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.977416    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"
Oct 30 19:42:33.977448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.977436    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879" event={"ID":"7a0e0a33-294b-4a98-baeb-65d299391c17","Type":"ContainerDied","Data":"fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5"}
Oct 30 19:42:33.977685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:33.977483    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf1e790528fbcd780ef0a0b36369c661bea291792276247c735bff9c4caf4e5"
Oct 30 19:42:45.970375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:45.970342    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:42:45.970874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:45.970860    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7a0e0a33-294b-4a98-baeb-65d299391c17" containerName="revoke-keys"
Oct 30 19:42:45.970939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:45.970875    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0e0a33-294b-4a98-baeb-65d299391c17" containerName="revoke-keys"
Oct 30 19:42:45.971002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:45.970992    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0e0a33-294b-4a98-baeb-65d299391c17" containerName="revoke-keys"
Oct 30 19:42:45.974423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:45.974406    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.024436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.024401    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b0a183-openshift-default-dockercfg-hjv6k\""
Oct 30 19:42:46.092025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.091985    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bzcp\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092233    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092422 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092399    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092491    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092667    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092776 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092756    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.092829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092815    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.093389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.092842    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.124652 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.124612    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:42:46.197789 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.197744    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.197981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.197799    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.197981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.197869    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.197981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.197919    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.197981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.197980    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.198210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.198007    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.198210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.198053    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bzcp\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.198210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.198116    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.198210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.198154    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.198778 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.198749    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.199788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.199760    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.200796 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.200721    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.201498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.201180    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.201498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.201448    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.202933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.202902    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.203890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.203872    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.257127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.257066    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.259008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.258962    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bzcp\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.289703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.289669    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:46.616918 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.616838    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:42:46.712955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.712922    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:42:46.714445 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:42:46.714418    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49411fe_188a_4a92_896e_bb2dcdc1e302.slice/crio-468315be26af4553c770a4a4cf8f5bc274e1866aa83dced253f918aa8a2ef171 WatchSource:0}: Error finding container 468315be26af4553c770a4a4cf8f5bc274e1866aa83dced253f918aa8a2ef171: Status 404 returned error can't find the container with id 468315be26af4553c770a4a4cf8f5bc274e1866aa83dced253f918aa8a2ef171
Oct 30 19:42:46.716868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.716837    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:42:46.716963 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.716900    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:42:46.716963 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:46.716929    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:42:47.066066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.065985    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" event={"ID":"e49411fe-188a-4a92-896e-bb2dcdc1e302","Type":"ContainerStarted","Data":"0218a0934c7f59a36533f969a6c3f39c47c0cbd0020e366cf694a05fd5f31aca"}
Oct 30 19:42:47.066066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.066042    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" event={"ID":"e49411fe-188a-4a92-896e-bb2dcdc1e302","Type":"ContainerStarted","Data":"468315be26af4553c770a4a4cf8f5bc274e1866aa83dced253f918aa8a2ef171"}
Oct 30 19:42:47.183252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.183188    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podStartSLOduration=2.183169071 podStartE2EDuration="2.183169071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:47.182428943 +0000 UTC m=+1076.686832689" watchObservedRunningTime="2026-10-30 19:42:47.183169071 +0000 UTC m=+1076.687572819"
Oct 30 19:42:47.290041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.290007    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:47.291606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.291545    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:47.291738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.291636    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:47.426259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.425772    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"]
Oct 30 19:42:47.431689 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.430858    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.484527 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.484475    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:42:47.484709 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.484615    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:42:47.486719 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.486691    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"]
Oct 30 19:42:47.618632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.618598    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.618815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.618652    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nwc\" (UniqueName: \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.618815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.618693    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.720509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.720432    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.720509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.720489    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nwc\" (UniqueName: \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.720720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.720529    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.721325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.721305    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.722951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.722920    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:47.782397 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:47.782361    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nwc\" (UniqueName: \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\") pod \"maas-api-revoke-sub-5d8b269d-fck75\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:48.054931 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:48.054847    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:48.290362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:48.290326    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:48.290986 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:48.290392    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:48.478023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:48.477972    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"]
Oct 30 19:42:48.487078 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:42:48.487049    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bbb91a_dcd7_4ebd_9af8_bb2dd1a04013.slice/crio-849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da WatchSource:0}: Error finding container 849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da: Status 404 returned error can't find the container with id 849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da
Oct 30 19:42:49.098704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:49.098655    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75" event={"ID":"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013","Type":"ContainerStarted","Data":"59dfeaa51a2d6ea71d34015d6aeba2f96038931e602e64bb6abcf27ebd83e92a"}
Oct 30 19:42:49.098878 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:49.098762    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75" event={"ID":"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013","Type":"ContainerStarted","Data":"849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da"}
Oct 30 19:42:49.211526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:49.211471    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75" podStartSLOduration=2.211456161 podStartE2EDuration="2.211456161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:49.209455783 +0000 UTC m=+1078.713859533" watchObservedRunningTime="2026-10-30 19:42:49.211456161 +0000 UTC m=+1078.715859909"
Oct 30 19:42:49.290350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:49.290312    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:49.290523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:49.290371    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:50.112041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:50.112002    5085 generic.go:356] "Generic (PLEG): container finished" podID="81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" containerID="59dfeaa51a2d6ea71d34015d6aeba2f96038931e602e64bb6abcf27ebd83e92a" exitCode=0
Oct 30 19:42:50.112212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:50.112064    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75" event={"ID":"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013","Type":"ContainerDied","Data":"59dfeaa51a2d6ea71d34015d6aeba2f96038931e602e64bb6abcf27ebd83e92a"}
Oct 30 19:42:50.291272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:50.291237    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:50.291828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:50.291289    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:51.290901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.290870    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:51.291045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.290933    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:51.322460 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.322433    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:51.391358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.390779    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\") pod \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") "
Oct 30 19:42:51.393659 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.391641    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\" (UniqueName: \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\") pod \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") "
Oct 30 19:42:51.393923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.393904    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\") pod \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\" (UID: \"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013\") "
Oct 30 19:42:51.394401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.394372    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca" pod "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" (UID: "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:51.394898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.394859    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.396676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.396652    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca" pod "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" (UID: "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:51.397005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.396980    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc" pod "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" (UID: "81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013"). InnerVolumeSpecName "kube-api-access-q2nwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:51.496173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.496125    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.496648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:51.496622    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nwc\" (UniqueName: \"kubernetes.io/projected/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013-kube-api-access-q2nwc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:51.940662 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:42:51.940620    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:42:52.126433 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:52.126407    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"
Oct 30 19:42:52.126621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:52.126443    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75" event={"ID":"81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013","Type":"ContainerDied","Data":"849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da"}
Oct 30 19:42:52.126621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:52.126503    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="849c780e44f3993cde07583b005de01cba8078f64e7535d54e202e53df5e02da"
Oct 30 19:42:52.291082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:52.290994    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:42:52.291082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:52.291058    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:42:53.500534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:53.500493    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:53.501708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:53.501686    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:42:53.502712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:42:53.502694    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:43:01.374885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.374849    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"]
Oct 30 19:43:01.375466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.375446    5085 cpu_manager.go:414] "containerMap: removing container" podUID="81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" containerName="revoke-keys"
Oct 30 19:43:01.375466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.375466    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" containerName="revoke-keys"
Oct 30 19:43:01.375610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.375572    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" containerName="revoke-keys"
Oct 30 19:43:01.385846 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.385818    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.437385 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.437351    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:01.442324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.442294    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"]
Oct 30 19:43:01.484309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.484283    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:43:01.509982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.509951    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.510137 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.510005    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.510137 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.510124    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxc9t\" (UniqueName: \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.611518 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.611487    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxc9t\" (UniqueName: \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.611722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.611608    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.611722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.611658    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.612310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.612284    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.614024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.614001    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.669975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.669947    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxc9t\" (UniqueName: \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\") pod \"maas-api-revoke-sub-39879b3d-x5b5f\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:01.702978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:01.702937    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:02.072770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.072738    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"]
Oct 30 19:43:02.074015 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:43:02.073988    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b88d4f_b55f_4f21_9690_95b4b7b0555b.slice/crio-89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222 WatchSource:0}: Error finding container 89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222: Status 404 returned error can't find the container with id 89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222
Oct 30 19:43:02.084989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.084961    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"]
Oct 30 19:43:02.090453 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.090429    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.178022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.177985    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"]
Oct 30 19:43:02.199277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.199239    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f" event={"ID":"c9b88d4f-b55f-4f21-9690-95b4b7b0555b","Type":"ContainerStarted","Data":"89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222"}
Oct 30 19:43:02.220057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.218350    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92smg\" (UniqueName: \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.220057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.218401    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.220057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.218650    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.319870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.319839    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.320047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.319946    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92smg\" (UniqueName: \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.320047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.319977    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.320499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.320476    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.322339 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.322294    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.383133 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.383107    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92smg\" (UniqueName: \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\") pod \"maas-api-revoke-sub-1a1b59fa-x5s7b\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.421403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.421376    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:02.796908 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:02.796880    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"]
Oct 30 19:43:03.206828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:03.206791    5085 generic.go:356] "Generic (PLEG): container finished" podID="c9b88d4f-b55f-4f21-9690-95b4b7b0555b" containerID="f921f4944a6aa445ccc65da2531999795496850d61ae075bbf03197b4c27365f" exitCode=0
Oct 30 19:43:03.207027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:03.206876    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f" event={"ID":"c9b88d4f-b55f-4f21-9690-95b4b7b0555b","Type":"ContainerDied","Data":"f921f4944a6aa445ccc65da2531999795496850d61ae075bbf03197b4c27365f"}
Oct 30 19:43:03.208475 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:03.208450    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b" event={"ID":"79ae6626-147a-4872-8088-0a7ae45ee83d","Type":"ContainerStarted","Data":"614fd194334fa07ef6f510e4a2141b3a1790b2f1bb61e0dd79b7ae0dae3b52ab"}
Oct 30 19:43:03.208603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:03.208488    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b" event={"ID":"79ae6626-147a-4872-8088-0a7ae45ee83d","Type":"ContainerStarted","Data":"5c4671916dc5a8452e5a5bbbbee7a3c6ad69a129c029094ba8c7f69cb9063f41"}
Oct 30 19:43:03.433450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:03.433384    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b" podStartSLOduration=1.433365011 podStartE2EDuration="1.433365011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:03.428549374 +0000 UTC m=+1092.932953147" watchObservedRunningTime="2026-10-30 19:43:03.433365011 +0000 UTC m=+1092.937768760"
Oct 30 19:43:04.216128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.216092    5085 generic.go:356] "Generic (PLEG): container finished" podID="79ae6626-147a-4872-8088-0a7ae45ee83d" containerID="614fd194334fa07ef6f510e4a2141b3a1790b2f1bb61e0dd79b7ae0dae3b52ab" exitCode=0
Oct 30 19:43:04.216304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.216156    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b" event={"ID":"79ae6626-147a-4872-8088-0a7ae45ee83d","Type":"ContainerDied","Data":"614fd194334fa07ef6f510e4a2141b3a1790b2f1bb61e0dd79b7ae0dae3b52ab"}
Oct 30 19:43:04.377354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.377326    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:04.443253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.443223    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\" (UniqueName: \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\") pod \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") "
Oct 30 19:43:04.443638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.443294    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\") pod \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") "
Oct 30 19:43:04.443638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.443446    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\") pod \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\" (UID: \"c9b88d4f-b55f-4f21-9690-95b4b7b0555b\") "
Oct 30 19:43:04.443871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.443852    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca" pod "c9b88d4f-b55f-4f21-9690-95b4b7b0555b" (UID: "c9b88d4f-b55f-4f21-9690-95b4b7b0555b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:04.445353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.445328    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca" pod "c9b88d4f-b55f-4f21-9690-95b4b7b0555b" (UID: "c9b88d4f-b55f-4f21-9690-95b4b7b0555b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:04.445495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.445398    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t" pod "c9b88d4f-b55f-4f21-9690-95b4b7b0555b" (UID: "c9b88d4f-b55f-4f21-9690-95b4b7b0555b"). InnerVolumeSpecName "kube-api-access-gxc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:04.545327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.545229    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxc9t\" (UniqueName: \"kubernetes.io/projected/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-kube-api-access-gxc9t\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:04.545327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.545272    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:04.545327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:04.545282    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9b88d4f-b55f-4f21-9690-95b4b7b0555b-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.228180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.228137    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f" event={"ID":"c9b88d4f-b55f-4f21-9690-95b4b7b0555b","Type":"ContainerDied","Data":"89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222"}
Oct 30 19:43:05.228180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.228166    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"
Oct 30 19:43:05.228418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.228226    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89bd1ab8329bfd32a37d5b24ad41fdf3a5b9a9f83288a199f131f812667ba222"
Oct 30 19:43:05.394536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.394511    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:05.455816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.455785    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\") pod \"79ae6626-147a-4872-8088-0a7ae45ee83d\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") "
Oct 30 19:43:05.456295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.455871    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\") pod \"79ae6626-147a-4872-8088-0a7ae45ee83d\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") "
Oct 30 19:43:05.456295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.455968    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\" (UniqueName: \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\") pod \"79ae6626-147a-4872-8088-0a7ae45ee83d\" (UID: \"79ae6626-147a-4872-8088-0a7ae45ee83d\") "
Oct 30 19:43:05.456295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.456179    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca" pod "79ae6626-147a-4872-8088-0a7ae45ee83d" (UID: "79ae6626-147a-4872-8088-0a7ae45ee83d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:05.456450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.456431    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.458002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.457971    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca" pod "79ae6626-147a-4872-8088-0a7ae45ee83d" (UID: "79ae6626-147a-4872-8088-0a7ae45ee83d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:05.458107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.458026    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg" pod "79ae6626-147a-4872-8088-0a7ae45ee83d" (UID: "79ae6626-147a-4872-8088-0a7ae45ee83d"). InnerVolumeSpecName "kube-api-access-92smg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:05.557094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.557017    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92smg\" (UniqueName: \"kubernetes.io/projected/79ae6626-147a-4872-8088-0a7ae45ee83d-kube-api-access-92smg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:05.557094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:05.557047    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79ae6626-147a-4872-8088-0a7ae45ee83d-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:06.235821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:06.235787    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b" event={"ID":"79ae6626-147a-4872-8088-0a7ae45ee83d","Type":"ContainerDied","Data":"5c4671916dc5a8452e5a5bbbbee7a3c6ad69a129c029094ba8c7f69cb9063f41"}
Oct 30 19:43:06.235821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:06.235808    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"
Oct 30 19:43:06.236027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:06.235835    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4671916dc5a8452e5a5bbbbee7a3c6ad69a129c029094ba8c7f69cb9063f41"
Oct 30 19:43:09.255548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:09.255515    5085 generic.go:356] "Generic (PLEG): container finished" podID="f34d47fb-addd-4b47-8692-d378137ffac1" containerID="56861f3ebdc7885a2d018aeef4fccb722a7c64a861aba3ed8d2cb8323d29f88e" exitCode=0
Oct 30 19:43:09.255957 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:09.255591    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5qjs-nmczz" event={"ID":"f34d47fb-addd-4b47-8692-d378137ffac1","Type":"ContainerDied","Data":"56861f3ebdc7885a2d018aeef4fccb722a7c64a861aba3ed8d2cb8323d29f88e"}
Oct 30 19:43:10.418529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.418501    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:43:10.515273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515193    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\" (UniqueName: \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\") pod \"f34d47fb-addd-4b47-8692-d378137ffac1\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") "
Oct 30 19:43:10.515273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515266    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\") pod \"f34d47fb-addd-4b47-8692-d378137ffac1\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") "
Oct 30 19:43:10.515465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515322    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\") pod \"f34d47fb-addd-4b47-8692-d378137ffac1\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") "
Oct 30 19:43:10.515541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515524    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\") pod \"f34d47fb-addd-4b47-8692-d378137ffac1\" (UID: \"f34d47fb-addd-4b47-8692-d378137ffac1\") "
Oct 30 19:43:10.515727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515699    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle" pod "f34d47fb-addd-4b47-8692-d378137ffac1" (UID: "f34d47fb-addd-4b47-8692-d378137ffac1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:10.515848 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515744    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle" pod "f34d47fb-addd-4b47-8692-d378137ffac1" (UID: "f34d47fb-addd-4b47-8692-d378137ffac1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:10.515915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.515846    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path" pod "f34d47fb-addd-4b47-8692-d378137ffac1" (UID: "f34d47fb-addd-4b47-8692-d378137ffac1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:10.516110 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.516092    5085 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f34d47fb-addd-4b47-8692-d378137ffac1-archives-path\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:10.516174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.516116    5085 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-service-ca-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:10.516174 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.516131    5085 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f34d47fb-addd-4b47-8692-d378137ffac1-trusted-ca-bundle\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:10.517437 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.517416    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj" pod "f34d47fb-addd-4b47-8692-d378137ffac1" (UID: "f34d47fb-addd-4b47-8692-d378137ffac1"). InnerVolumeSpecName "kube-api-access-7tqvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:10.617282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:10.617247    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tqvj\" (UniqueName: \"kubernetes.io/projected/f34d47fb-addd-4b47-8692-d378137ffac1-kube-api-access-7tqvj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:11.269598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:11.269571    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5qjs-nmczz"
Oct 30 19:43:11.269768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:11.269611    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5qjs-nmczz" event={"ID":"f34d47fb-addd-4b47-8692-d378137ffac1","Type":"ContainerDied","Data":"daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4"}
Oct 30 19:43:11.269768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:11.269655    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daef695a7360e791b369ccb1048a0dcd4e9d6c92e5ba22be16b6f0403835beb4"
Oct 30 19:43:12.077930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:12.077888    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:43:16.850380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.850346    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"]
Oct 30 19:43:16.850981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.850958    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f34d47fb-addd-4b47-8692-d378137ffac1" containerName="insights-gathering"
Oct 30 19:43:16.851058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.850981    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34d47fb-addd-4b47-8692-d378137ffac1" containerName="insights-gathering"
Oct 30 19:43:16.851058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.850997    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c9b88d4f-b55f-4f21-9690-95b4b7b0555b" containerName="revoke-keys"
Oct 30 19:43:16.851058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851002    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b88d4f-b55f-4f21-9690-95b4b7b0555b" containerName="revoke-keys"
Oct 30 19:43:16.851058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851020    5085 cpu_manager.go:414] "containerMap: removing container" podUID="79ae6626-147a-4872-8088-0a7ae45ee83d" containerName="revoke-keys"
Oct 30 19:43:16.851058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851025    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ae6626-147a-4872-8088-0a7ae45ee83d" containerName="revoke-keys"
Oct 30 19:43:16.851204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851102    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b88d4f-b55f-4f21-9690-95b4b7b0555b" containerName="revoke-keys"
Oct 30 19:43:16.851204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851113    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ae6626-147a-4872-8088-0a7ae45ee83d" containerName="revoke-keys"
Oct 30 19:43:16.851204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.851123    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34d47fb-addd-4b47-8692-d378137ffac1" containerName="insights-gathering"
Oct 30 19:43:16.854819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.854803    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:16.953646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.953617    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:16.953822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.953784    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:43:16.957599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:16.957544    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"]
Oct 30 19:43:17.025512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.025481    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.025512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.025516    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6zff\" (UniqueName: \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.025743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.025724    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.127414 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.127338    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.127556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.127441    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.127556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.127460    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6zff\" (UniqueName: \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.128105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.128079    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.129622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.129601    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.186386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.186348    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6zff\" (UniqueName: \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\") pod \"maas-api-revoke-sub-25916885-9pdd2\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.469502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.469471    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:17.843621 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:17.843587    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"]
Oct 30 19:43:17.844492 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:43:17.844457    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c317712_7d83_4e7d_9ad1_78a9af22da5e.slice/crio-4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364 WatchSource:0}: Error finding container 4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364: Status 404 returned error can't find the container with id 4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364
Oct 30 19:43:18.357164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:18.357125    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2" event={"ID":"7c317712-7d83-4e7d-9ad1-78a9af22da5e","Type":"ContainerStarted","Data":"9d00d6393e756093ed34e0f274db77f773e54fc0ff0f99cbaf1da1a971324b55"}
Oct 30 19:43:18.357620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:18.357189    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2" event={"ID":"7c317712-7d83-4e7d-9ad1-78a9af22da5e","Type":"ContainerStarted","Data":"4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364"}
Oct 30 19:43:19.365289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:19.365254    5085 generic.go:356] "Generic (PLEG): container finished" podID="7c317712-7d83-4e7d-9ad1-78a9af22da5e" containerID="9d00d6393e756093ed34e0f274db77f773e54fc0ff0f99cbaf1da1a971324b55" exitCode=0
Oct 30 19:43:19.366074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:19.365345    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2" event={"ID":"7c317712-7d83-4e7d-9ad1-78a9af22da5e","Type":"ContainerDied","Data":"9d00d6393e756093ed34e0f274db77f773e54fc0ff0f99cbaf1da1a971324b55"}
Oct 30 19:43:20.519172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.519143    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:20.666122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.666048    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\" (UniqueName: \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\") pod \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") "
Oct 30 19:43:20.666294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.666132    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\") pod \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") "
Oct 30 19:43:20.666369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.666293    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\") pod \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\" (UID: \"7c317712-7d83-4e7d-9ad1-78a9af22da5e\") "
Oct 30 19:43:20.666763 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.666722    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca" pod "7c317712-7d83-4e7d-9ad1-78a9af22da5e" (UID: "7c317712-7d83-4e7d-9ad1-78a9af22da5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:20.668192 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.668158    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca" pod "7c317712-7d83-4e7d-9ad1-78a9af22da5e" (UID: "7c317712-7d83-4e7d-9ad1-78a9af22da5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:20.668636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.668615    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff" pod "7c317712-7d83-4e7d-9ad1-78a9af22da5e" (UID: "7c317712-7d83-4e7d-9ad1-78a9af22da5e"). InnerVolumeSpecName "kube-api-access-q6zff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:20.767617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.767551    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6zff\" (UniqueName: \"kubernetes.io/projected/7c317712-7d83-4e7d-9ad1-78a9af22da5e-kube-api-access-q6zff\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:20.767617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.767618    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:20.767833 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:20.767631    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c317712-7d83-4e7d-9ad1-78a9af22da5e-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:21.386555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:21.386073    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2" event={"ID":"7c317712-7d83-4e7d-9ad1-78a9af22da5e","Type":"ContainerDied","Data":"4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364"}
Oct 30 19:43:21.386555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:21.386153    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f1bee4babda4a2cb44d861ce9aa5ddcb03ff7e729a7d0bbb3010d8068c30364"
Oct 30 19:43:21.386555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:21.386243    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"
Oct 30 19:43:21.939302 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:43:21.939268    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:43:22.045777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.045740    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"]
Oct 30 19:43:22.046294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.046279    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7c317712-7d83-4e7d-9ad1-78a9af22da5e" containerName="revoke-keys"
Oct 30 19:43:22.046342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.046296    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c317712-7d83-4e7d-9ad1-78a9af22da5e" containerName="revoke-keys"
Oct 30 19:43:22.046406 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.046396    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c317712-7d83-4e7d-9ad1-78a9af22da5e" containerName="revoke-keys"
Oct 30 19:43:22.049982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.049961    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.101670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.101052    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:22.105679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.105650    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"]
Oct 30 19:43:22.148047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.148009    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:43:22.183797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.183753    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.183958 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.183889    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.184059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.184042    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spcv2\" (UniqueName: \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.285309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.285226    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spcv2\" (UniqueName: \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.285309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.285284    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.285542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.285339    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.286051 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.286028    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.287706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.287689    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.344229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.344203    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spcv2\" (UniqueName: \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\") pod \"maas-api-revoke-sub-2babaa97-n2hr2\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.363449 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.363423    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:22.738516 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:22.738486    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"]
Oct 30 19:43:22.739766 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:43:22.739743    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6670f79d_9ad6_451e_8e57_eb5897307d6d.slice/crio-2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a WatchSource:0}: Error finding container 2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a: Status 404 returned error can't find the container with id 2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a
Oct 30 19:43:23.416856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:23.416746    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2" event={"ID":"6670f79d-9ad6-451e-8e57-eb5897307d6d","Type":"ContainerStarted","Data":"ffcdff32950f78a83119feba4a9b4a4f4cda6f154bc0cd830902f36f3bf5a0c5"}
Oct 30 19:43:23.416856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:23.416820    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2" event={"ID":"6670f79d-9ad6-451e-8e57-eb5897307d6d","Type":"ContainerStarted","Data":"2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a"}
Oct 30 19:43:23.529017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:23.528963    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2" podStartSLOduration=1.528948921 podStartE2EDuration="1.528948921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:23.526690783 +0000 UTC m=+1113.031094529" watchObservedRunningTime="2026-10-30 19:43:23.528948921 +0000 UTC m=+1113.033352724"
Oct 30 19:43:24.424660 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:24.424619    5085 generic.go:356] "Generic (PLEG): container finished" podID="6670f79d-9ad6-451e-8e57-eb5897307d6d" containerID="ffcdff32950f78a83119feba4a9b4a4f4cda6f154bc0cd830902f36f3bf5a0c5" exitCode=0
Oct 30 19:43:24.425082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:24.424703    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2" event={"ID":"6670f79d-9ad6-451e-8e57-eb5897307d6d","Type":"ContainerDied","Data":"ffcdff32950f78a83119feba4a9b4a4f4cda6f154bc0cd830902f36f3bf5a0c5"}
Oct 30 19:43:25.588487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.588456    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:25.719353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.719273    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\") pod \"6670f79d-9ad6-451e-8e57-eb5897307d6d\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") "
Oct 30 19:43:25.719353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.719347    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\") pod \"6670f79d-9ad6-451e-8e57-eb5897307d6d\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") "
Oct 30 19:43:25.719586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.719447    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\" (UniqueName: \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\") pod \"6670f79d-9ad6-451e-8e57-eb5897307d6d\" (UID: \"6670f79d-9ad6-451e-8e57-eb5897307d6d\") "
Oct 30 19:43:25.719760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.719730    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca" pod "6670f79d-9ad6-451e-8e57-eb5897307d6d" (UID: "6670f79d-9ad6-451e-8e57-eb5897307d6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:25.719965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.719945    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:25.721504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.721477    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca" pod "6670f79d-9ad6-451e-8e57-eb5897307d6d" (UID: "6670f79d-9ad6-451e-8e57-eb5897307d6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:25.721744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.721719    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2" pod "6670f79d-9ad6-451e-8e57-eb5897307d6d" (UID: "6670f79d-9ad6-451e-8e57-eb5897307d6d"). InnerVolumeSpecName "kube-api-access-spcv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:25.821153 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.821117    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6670f79d-9ad6-451e-8e57-eb5897307d6d-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:25.821153 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:25.821147    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spcv2\" (UniqueName: \"kubernetes.io/projected/6670f79d-9ad6-451e-8e57-eb5897307d6d-kube-api-access-spcv2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:26.440592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:26.440544    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"
Oct 30 19:43:26.440592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:26.440579    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2" event={"ID":"6670f79d-9ad6-451e-8e57-eb5897307d6d","Type":"ContainerDied","Data":"2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a"}
Oct 30 19:43:26.440821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:26.440657    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3ceeb2e0bdc14b963403f43cdf3b6f1e28e5e93ec50b6adf9dc63711fb565a"
Oct 30 19:43:32.849460 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.849375    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"]
Oct 30 19:43:32.849993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.849938    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6670f79d-9ad6-451e-8e57-eb5897307d6d" containerName="revoke-keys"
Oct 30 19:43:32.849993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.849953    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6670f79d-9ad6-451e-8e57-eb5897307d6d" containerName="revoke-keys"
Oct 30 19:43:32.850083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.850054    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6670f79d-9ad6-451e-8e57-eb5897307d6d" containerName="revoke-keys"
Oct 30 19:43:32.853524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.853506    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:32.908396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.908367    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"]
Oct 30 19:43:32.910235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.910207    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:32.910351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.910262    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:32.910407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.910347    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqdq\" (UniqueName: \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:32.951122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.951093    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:32.951286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:32.951272    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:43:33.011781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.011747    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.011950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.011806    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.011950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.011858    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqdq\" (UniqueName: \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.012480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.012454    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.014171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.014144    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.071128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.071097    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqdq\" (UniqueName: \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\") pod \"maas-api-revoke-sub-80269d8c-6wx5d\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.168068 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.167995    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:33.544383 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:33.543478    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"]
Oct 30 19:43:33.547094 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:43:33.547067    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b560f1_9897_42a0_aa8e_71a1809835cf.slice/crio-84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee WatchSource:0}: Error finding container 84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee: Status 404 returned error can't find the container with id 84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee
Oct 30 19:43:34.542866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:34.542832    5085 generic.go:356] "Generic (PLEG): container finished" podID="94b560f1-9897-42a0-aa8e-71a1809835cf" containerID="68adcf469c80d24599286542d65d2630c1f03a0c185c43c815612baf4169c0ea" exitCode=0
Oct 30 19:43:34.543241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:34.542878    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d" event={"ID":"94b560f1-9897-42a0-aa8e-71a1809835cf","Type":"ContainerDied","Data":"68adcf469c80d24599286542d65d2630c1f03a0c185c43c815612baf4169c0ea"}
Oct 30 19:43:34.543241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:34.542954    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d" event={"ID":"94b560f1-9897-42a0-aa8e-71a1809835cf","Type":"ContainerStarted","Data":"84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee"}
Oct 30 19:43:35.691849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.691819    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:35.781774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.781734    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\") pod \"94b560f1-9897-42a0-aa8e-71a1809835cf\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") "
Oct 30 19:43:35.781950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.781791    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\" (UniqueName: \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\") pod \"94b560f1-9897-42a0-aa8e-71a1809835cf\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") "
Oct 30 19:43:35.781950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.781834    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\") pod \"94b560f1-9897-42a0-aa8e-71a1809835cf\" (UID: \"94b560f1-9897-42a0-aa8e-71a1809835cf\") "
Oct 30 19:43:35.782218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.782192    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca" pod "94b560f1-9897-42a0-aa8e-71a1809835cf" (UID: "94b560f1-9897-42a0-aa8e-71a1809835cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:35.783975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.783950    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca" pod "94b560f1-9897-42a0-aa8e-71a1809835cf" (UID: "94b560f1-9897-42a0-aa8e-71a1809835cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:35.784075 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.784021    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq" pod "94b560f1-9897-42a0-aa8e-71a1809835cf" (UID: "94b560f1-9897-42a0-aa8e-71a1809835cf"). InnerVolumeSpecName "kube-api-access-2sqdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:35.883602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.883500    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:35.883602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.883529    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sqdq\" (UniqueName: \"kubernetes.io/projected/94b560f1-9897-42a0-aa8e-71a1809835cf-kube-api-access-2sqdq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:35.883602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:35.883542    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94b560f1-9897-42a0-aa8e-71a1809835cf-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:36.558896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:36.558861    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d" event={"ID":"94b560f1-9897-42a0-aa8e-71a1809835cf","Type":"ContainerDied","Data":"84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee"}
Oct 30 19:43:36.559053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:36.558929    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84cb7f7e721fefcefd79f4104500f2447ce37c0c8c9d1c018d68f90644a2abee"
Oct 30 19:43:36.559053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:36.558875    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"
Oct 30 19:43:48.600018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:48.599980    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:43:51.939378 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:43:51.939335    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:43:53.728253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.728219    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"]
Oct 30 19:43:53.728790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.728776    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94b560f1-9897-42a0-aa8e-71a1809835cf" containerName="revoke-keys"
Oct 30 19:43:53.728845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.728791    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b560f1-9897-42a0-aa8e-71a1809835cf" containerName="revoke-keys"
Oct 30 19:43:53.728927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.728917    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b560f1-9897-42a0-aa8e-71a1809835cf" containerName="revoke-keys"
Oct 30 19:43:53.733714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.733691    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:53.783801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.783763    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:43:53.830321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.830284    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:43:53.833741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.833711    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"]
Oct 30 19:43:53.913913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.913878    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnd8\" (UniqueName: \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:53.914089 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.913918    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:53.914141 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:53.914077    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.015715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.015627    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.015902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.015749    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnd8\" (UniqueName: \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.015902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.015771    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.016415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.016395    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.018084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.018059    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.074015 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.073983    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnd8\" (UniqueName: \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\") pod \"maas-api-revoke-sub-6279f074-gmvxf\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.349439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.349360    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:54.742827 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:54.742780    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"]
Oct 30 19:43:55.693593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:55.693543    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf" event={"ID":"eca9faf3-066a-4694-9342-edb682351f5f","Type":"ContainerStarted","Data":"e71dabfc9fe4eb4fb6f1351a5f3037b0b81c175244969d46878d999e69bbae20"}
Oct 30 19:43:55.693785 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:55.693616    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf" event={"ID":"eca9faf3-066a-4694-9342-edb682351f5f","Type":"ContainerStarted","Data":"ee668366883ba4769b5612a019679c68a4d4797f972f4c3383d20b9e883a1984"}
Oct 30 19:43:55.806031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:55.805978    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf" podStartSLOduration=2.805962544 podStartE2EDuration="2.805962544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:55.804265241 +0000 UTC m=+1145.308668988" watchObservedRunningTime="2026-10-30 19:43:55.805962544 +0000 UTC m=+1145.310366291"
Oct 30 19:43:56.702396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:56.702361    5085 generic.go:356] "Generic (PLEG): container finished" podID="eca9faf3-066a-4694-9342-edb682351f5f" containerID="e71dabfc9fe4eb4fb6f1351a5f3037b0b81c175244969d46878d999e69bbae20" exitCode=0
Oct 30 19:43:56.702638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:56.702454    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf" event={"ID":"eca9faf3-066a-4694-9342-edb682351f5f","Type":"ContainerDied","Data":"e71dabfc9fe4eb4fb6f1351a5f3037b0b81c175244969d46878d999e69bbae20"}
Oct 30 19:43:57.882282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.882258    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:57.985612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.985484    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\" (UniqueName: \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\") pod \"eca9faf3-066a-4694-9342-edb682351f5f\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") "
Oct 30 19:43:57.985807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.985688    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\") pod \"eca9faf3-066a-4694-9342-edb682351f5f\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") "
Oct 30 19:43:57.985807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.985745    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\") pod \"eca9faf3-066a-4694-9342-edb682351f5f\" (UID: \"eca9faf3-066a-4694-9342-edb682351f5f\") "
Oct 30 19:43:57.986093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.986066    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca" pod "eca9faf3-066a-4694-9342-edb682351f5f" (UID: "eca9faf3-066a-4694-9342-edb682351f5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:57.987694 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.987668    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8" pod "eca9faf3-066a-4694-9342-edb682351f5f" (UID: "eca9faf3-066a-4694-9342-edb682351f5f"). InnerVolumeSpecName "kube-api-access-5cnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:57.987822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:57.987798    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca" pod "eca9faf3-066a-4694-9342-edb682351f5f" (UID: "eca9faf3-066a-4694-9342-edb682351f5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:58.087198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.087158    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnd8\" (UniqueName: \"kubernetes.io/projected/eca9faf3-066a-4694-9342-edb682351f5f-kube-api-access-5cnd8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:58.087198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.087195    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:58.087198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.087208    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9faf3-066a-4694-9342-edb682351f5f-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:58.717404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.717371    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"
Oct 30 19:43:58.717404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.717397    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf" event={"ID":"eca9faf3-066a-4694-9342-edb682351f5f","Type":"ContainerDied","Data":"ee668366883ba4769b5612a019679c68a4d4797f972f4c3383d20b9e883a1984"}
Oct 30 19:43:58.717644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:43:58.717446    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee668366883ba4769b5612a019679c68a4d4797f972f4c3383d20b9e883a1984"
Oct 30 19:44:04.165432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.165388    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:44:04.166168 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.166136    5085 cpu_manager.go:414] "containerMap: removing container" podUID="eca9faf3-066a-4694-9342-edb682351f5f" containerName="revoke-keys"
Oct 30 19:44:04.166277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.166168    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca9faf3-066a-4694-9342-edb682351f5f" containerName="revoke-keys"
Oct 30 19:44:04.166373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.166358    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca9faf3-066a-4694-9342-edb682351f5f" containerName="revoke-keys"
Oct 30 19:44:04.169956 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.169926    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251003    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251057    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251143    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251188    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251214    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251250    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251290    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251382    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h427\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.251445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.251404    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.269896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.269865    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:44:04.352127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352096    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352148    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352182    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352212    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352257    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352288    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352328    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352435    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352594    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h427\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352836    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.352922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.352893    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.353053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.353033    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.353135 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.353106    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.353195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.353144    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.354815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.354791    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.355304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.355283    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.429633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.425226    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.449882 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.449841    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h427\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\") pod \"maas-default-gateway-openshift-default-694db55c87-tzs66\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:04.484554 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:04.484513    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:05.028981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.028697    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:44:05.032349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.030684    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:05.032349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.030767    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:05.032349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.030808    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:05.803374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.803329    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" event={"ID":"69148e2d-793c-4165-ab14-29013735444f","Type":"ContainerStarted","Data":"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"}
Oct 30 19:44:05.804010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.803405    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" event={"ID":"69148e2d-793c-4165-ab14-29013735444f","Type":"ContainerStarted","Data":"5bd3f394275c072fd23108f41e1aef0e837f72cbf24f2b18f33febc4af801505"}
Oct 30 19:44:05.966026 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:05.965969    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podStartSLOduration=1.965954668 podStartE2EDuration="1.965954668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:05.964685861 +0000 UTC m=+1155.469089611" watchObservedRunningTime="2026-10-30 19:44:05.965954668 +0000 UTC m=+1155.470358414"
Oct 30 19:44:06.490667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:06.490622    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:06.492839 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:06.492807    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:06.493004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:06.492865    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:07.485943 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:07.485908    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:07.486356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:07.485965    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:08.485431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:08.485391    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:08.485658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:08.485453    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:09.485929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:09.485892    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:09.486455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:09.485943    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:10.486080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:10.486039    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:10.486506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:10.486094    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:11.486163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:11.486127    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:11.486557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:11.486184    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:12.485268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:12.485220    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 19:44:12.485456 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:12.485290    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 19:44:13.952321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:13.951200    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:13.952321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:13.951981    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:13.953089 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:13.953062    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:44:14.174392 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:14.174353    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:44:19.299044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.299006    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:44:19.303411 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.303391    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.411701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.411663    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.411701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.411702    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mn6\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.411967 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.411733    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.411967 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.411914    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.412091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.411978    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.412091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.412063    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.412197 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.412132    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.412264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.412243    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.412324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.412284    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.505655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.505617    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:44:19.513191 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513162    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513227    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513279    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513299    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513338    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513362 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513357    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mn6\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513386    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513424    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513449    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.513698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.513675    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.514181 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.514154    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.514350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.514203    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.514459 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.514437    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.514701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.514666    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.516291 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.516270    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.517087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.517063    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.572822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.572747    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.573028 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.573004    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mn6\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\") pod \"maas-default-gateway-openshift-default-694db55c87-rmnvr\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:19.617688 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:19.617650    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:20.046440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.046403    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:44:20.047874 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:44:20.047848    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae7b2c8_2eed_4727_bb14_d0618dedca38.slice/crio-d4e601b1b57f642ee34d4aa5ae139514bd4834597c4712b620ae6c804f92a729 WatchSource:0}: Error finding container d4e601b1b57f642ee34d4aa5ae139514bd4834597c4712b620ae6c804f92a729: Status 404 returned error can't find the container with id d4e601b1b57f642ee34d4aa5ae139514bd4834597c4712b620ae6c804f92a729
Oct 30 19:44:20.050235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.050205    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:20.050304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.050272    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:20.050348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.050313    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:20.915531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.915488    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" event={"ID":"fae7b2c8-2eed-4727-bb14-d0618dedca38","Type":"ContainerStarted","Data":"73ed190d4775b380fa9544d20809b292b79f8e8d3912fc5e94e6d960043a8beb"}
Oct 30 19:44:20.915956 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:20.915554    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" event={"ID":"fae7b2c8-2eed-4727-bb14-d0618dedca38","Type":"ContainerStarted","Data":"d4e601b1b57f642ee34d4aa5ae139514bd4834597c4712b620ae6c804f92a729"}
Oct 30 19:44:21.130873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:21.130815    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podStartSLOduration=2.130803364 podStartE2EDuration="2.130803364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:21.083934973 +0000 UTC m=+1170.588338716" watchObservedRunningTime="2026-10-30 19:44:21.130803364 +0000 UTC m=+1170.635207111"
Oct 30 19:44:21.618641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:21.618605    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:21.620340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:21.620307    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:21.620455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:21.620364    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:21.940136 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:44:21.940101    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:44:22.618423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:22.618390    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:22.618634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:22.618440    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:23.618558 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:23.618519    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:23.619105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:23.618613    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:24.618990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:24.618955    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:24.619385 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:24.619008    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:25.618646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:25.618608    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:25.618829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:25.618667    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:26.619086 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:26.618984    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:26.619086 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:26.619057    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:27.618027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:27.617988    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 19:44:27.618238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:27.618083    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 19:44:28.622867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:28.622836    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:28.623351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:28.623235    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:28.630882 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:28.630862    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:44:34.412699 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.412665    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:44:34.416731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.416713    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.511555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.511514    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:44:34.520771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520738    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.520922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520786    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.520922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520850    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.520922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520892    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.521022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520947    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.521022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.520992    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.521088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.521034    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.521149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.521133    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvx6\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.521196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.521165    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622384    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622471    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvx6\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622496    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622586    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622616    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622636    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622679    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622699    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.622968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622718    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.623114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.622970    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.623160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.623134    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.623223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.623160    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.623293 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.623275    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.623526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.623503    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.624819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.624792    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.625216 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.625198    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.681085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.681049    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.681232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.681190    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvx6\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\") pod \"maas-default-gateway-openshift-default-694db55c87-4vfdh\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:34.733321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:34.733290    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:35.154935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:35.154896    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:44:35.155710 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:44:35.155682    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82afc92c_3c63_42d8_88cc_39b82e11f0f2.slice/crio-0c798e40adc149a405c2b74ff26fd1a27181f1bd98e58ef173973aa92f7567fa WatchSource:0}: Error finding container 0c798e40adc149a405c2b74ff26fd1a27181f1bd98e58ef173973aa92f7567fa: Status 404 returned error can't find the container with id 0c798e40adc149a405c2b74ff26fd1a27181f1bd98e58ef173973aa92f7567fa
Oct 30 19:44:35.158450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:35.158409    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:35.158536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:35.158483    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:35.158536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:35.158511    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:44:36.037038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.036996    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" event={"ID":"82afc92c-3c63-42d8-88cc-39b82e11f0f2","Type":"ContainerStarted","Data":"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"}
Oct 30 19:44:36.037448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.037064    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" event={"ID":"82afc92c-3c63-42d8-88cc-39b82e11f0f2","Type":"ContainerStarted","Data":"0c798e40adc149a405c2b74ff26fd1a27181f1bd98e58ef173973aa92f7567fa"}
Oct 30 19:44:36.202387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.202333    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podStartSLOduration=2.202317936 podStartE2EDuration="2.202317936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:36.198662546 +0000 UTC m=+1185.703066293" watchObservedRunningTime="2026-10-30 19:44:36.202317936 +0000 UTC m=+1185.706721683"
Oct 30 19:44:36.733917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.733879    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:36.735702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.735679    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.735790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:36.735727    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:37.734442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:37.734398    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.734991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:37.734460    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:38.734520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:38.734484    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.734950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:38.734551    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:39.734343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:39.734308    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.734526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:39.734362    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:40.734277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:40.734243    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.734700 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:40.734292    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:41.734103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:41.734061    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.734261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:41.734128    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:42.734172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:42.734134    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.734558 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:42.734200    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:44:43.737987 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:43.737955    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:43.738398 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:43.738232    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:43.739044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:43.739027    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:44:48.088623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:48.088580    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:44:50.770343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:50.770311    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:44:50.777898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:44:50.777871    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:44:51.939857 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:44:51.939812    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:45:00.231719 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.231618    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:00.245272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.245230    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:00.245447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.245369    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.249853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.249823    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:00.249983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.249953    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.254480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.254459    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:00.254720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.254545    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.296542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.296515    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.371587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.363035    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:45:00.439315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.439187    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449466    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449598    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9tc\" (UniqueName: \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449664    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449822    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449870    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449922    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stw9w\" (UniqueName: \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.449976    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcj2b\" (UniqueName: \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.450200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.450022    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpgr\" (UniqueName: \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.540645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.517748    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:00.547437 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.547154    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:00.551512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551479    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9tc\" (UniqueName: \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.551657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551587    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.551657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551646    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.551816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551676    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.551816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551740    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stw9w\" (UniqueName: \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.551816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551783    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcj2b\" (UniqueName: \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.552490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551824    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpgr\" (UniqueName: \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.552490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.551898    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.552828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.552804    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.553821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.553793    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.555607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.555532    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.556417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.556395    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.561722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.561636    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:00.633630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.630259    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcj2b\" (UniqueName: \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\") pod \"maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.634271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.634216    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpgr\" (UniqueName: \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:00.638982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.638928    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9tc\" (UniqueName: \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.639537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.639466    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stw9w\" (UniqueName: \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\") pod \"maas-api-key-cleanup-29889825-w4m8q\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.641258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.640325    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:00.644585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.644545    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:00.750020 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.748188    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:00.795700 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:00.795619    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:01.077014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.076982    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:01.078538 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:01.078509    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86db2ef4_1a34_4ca1_93d8_2ce8640f1a77.slice/crio-299b855389411d2e4b11e0e4945db9ee0efd665601b2bbeb6dd5f524f0315173 WatchSource:0}: Error finding container 299b855389411d2e4b11e0e4945db9ee0efd665601b2bbeb6dd5f524f0315173: Status 404 returned error can't find the container with id 299b855389411d2e4b11e0e4945db9ee0efd665601b2bbeb6dd5f524f0315173
Oct 30 19:45:01.121763 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.121736    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:01.122120 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:01.122095    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1988c7e9_48f2_466d_a343_3140593776f5.slice/crio-353b3920e5af37101cbf9701cf1466152e50c10aec0bc685b086e23c41ba32c9 WatchSource:0}: Error finding container 353b3920e5af37101cbf9701cf1466152e50c10aec0bc685b086e23c41ba32c9: Status 404 returned error can't find the container with id 353b3920e5af37101cbf9701cf1466152e50c10aec0bc685b086e23c41ba32c9
Oct 30 19:45:01.171889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.171865    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:01.182812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.182789    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:01.251899 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.251865    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerStarted","Data":"299b855389411d2e4b11e0e4945db9ee0efd665601b2bbeb6dd5f524f0315173"}
Oct 30 19:45:01.253111 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:01.253083    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerStarted","Data":"353b3920e5af37101cbf9701cf1466152e50c10aec0bc685b086e23c41ba32c9"}
Oct 30 19:45:01.259977 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:01.259946    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611bc7e5_26e1_4fe9_ae46_e8311819e931.slice/crio-d63473e962af28df9e1b2973e669568dcbd71f0c1bacb1652bcb161db6a9a976 WatchSource:0}: Error finding container d63473e962af28df9e1b2973e669568dcbd71f0c1bacb1652bcb161db6a9a976: Status 404 returned error can't find the container with id d63473e962af28df9e1b2973e669568dcbd71f0c1bacb1652bcb161db6a9a976
Oct 30 19:45:01.261287 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:01.261256    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753847fd_164e_4b4c_bd22_7ac53910d283.slice/crio-23bbfdc6fc6af73f6021950f2f5a753fe0cbb2811279eac6c355d687e4eafad5 WatchSource:0}: Error finding container 23bbfdc6fc6af73f6021950f2f5a753fe0cbb2811279eac6c355d687e4eafad5: Status 404 returned error can't find the container with id 23bbfdc6fc6af73f6021950f2f5a753fe0cbb2811279eac6c355d687e4eafad5
Oct 30 19:45:02.262132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.262097    5085 generic.go:356] "Generic (PLEG): container finished" podID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerID="1a643ffaf2443ebca90211cbed17c287c52ab1017ef1afcccedffa9ddb350e8b" exitCode=60
Oct 30 19:45:02.262533 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.262178    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerDied","Data":"1a643ffaf2443ebca90211cbed17c287c52ab1017ef1afcccedffa9ddb350e8b"}
Oct 30 19:45:02.262533 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.262526    5085 scope.go:122] "RemoveContainer" containerID="1a643ffaf2443ebca90211cbed17c287c52ab1017ef1afcccedffa9ddb350e8b"
Oct 30 19:45:02.264066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.263990    5085 generic.go:356] "Generic (PLEG): container finished" podID="1988c7e9-48f2-466d-a343-3140593776f5" containerID="fc18bdc7fda7dffaddc08a36dd649c159d892ac3bb4133e386d4b598eee3d40c" exitCode=60
Oct 30 19:45:02.264066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.264048    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerDied","Data":"fc18bdc7fda7dffaddc08a36dd649c159d892ac3bb4133e386d4b598eee3d40c"}
Oct 30 19:45:02.264302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.264286    5085 scope.go:122] "RemoveContainer" containerID="fc18bdc7fda7dffaddc08a36dd649c159d892ac3bb4133e386d4b598eee3d40c"
Oct 30 19:45:02.265972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.265949    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerStarted","Data":"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"}
Oct 30 19:45:02.266060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.265997    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerStarted","Data":"d63473e962af28df9e1b2973e669568dcbd71f0c1bacb1652bcb161db6a9a976"}
Oct 30 19:45:02.266127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.266085    5085 scope.go:122] "RemoveContainer" containerID="e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"
Oct 30 19:45:02.267639 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.267620    5085 generic.go:356] "Generic (PLEG): container finished" podID="753847fd-164e-4b4c-bd22-7ac53910d283" containerID="585cb1ee4bc48c0b2ea9b512b2fcb578fee2bea503e20d1312b72b8428170cf8" exitCode=60
Oct 30 19:45:02.267745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.267703    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerDied","Data":"585cb1ee4bc48c0b2ea9b512b2fcb578fee2bea503e20d1312b72b8428170cf8"}
Oct 30 19:45:02.267804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.267751    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerStarted","Data":"23bbfdc6fc6af73f6021950f2f5a753fe0cbb2811279eac6c355d687e4eafad5"}
Oct 30 19:45:02.267841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:02.267825    5085 scope.go:122] "RemoveContainer" containerID="585cb1ee4bc48c0b2ea9b512b2fcb578fee2bea503e20d1312b72b8428170cf8"
Oct 30 19:45:03.277617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.277584    5085 generic.go:356] "Generic (PLEG): container finished" podID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerID="3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431" exitCode=60
Oct 30 19:45:03.278027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.277592    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerDied","Data":"3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431"}
Oct 30 19:45:03.278027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.277749    5085 scope.go:122] "RemoveContainer" containerID="1a643ffaf2443ebca90211cbed17c287c52ab1017ef1afcccedffa9ddb350e8b"
Oct 30 19:45:03.278149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.278112    5085 scope.go:122] "RemoveContainer" containerID="3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431"
Oct 30 19:45:03.278443 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:03.278420    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7_odh-ai-gateway-infra(86db2ef4-1a34-4ca1-93d8-2ce8640f1a77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77"
Oct 30 19:45:03.280624 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.280600    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerStarted","Data":"73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"}
Oct 30 19:45:03.280917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.280889    5085 scope.go:122] "RemoveContainer" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"
Oct 30 19:45:03.281194 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:03.281173    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n_odh-ai-gateway-infra(1988c7e9-48f2-466d-a343-3140593776f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" podUID="1988c7e9-48f2-466d-a343-3140593776f5"
Oct 30 19:45:03.283481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.283456    5085 generic.go:356] "Generic (PLEG): container finished" podID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerID="e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3" exitCode=60
Oct 30 19:45:03.283481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.283477    5085 generic.go:356] "Generic (PLEG): container finished" podID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerID="db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71" exitCode=60
Oct 30 19:45:03.283671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.283536    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerDied","Data":"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"}
Oct 30 19:45:03.283671 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.283613    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerDied","Data":"db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71"}
Oct 30 19:45:03.283772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.283737    5085 scope.go:122] "RemoveContainer" containerID="db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71"
Oct 30 19:45:03.284020 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:03.283998    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-w4m8q_odh-ai-gateway-infra(611bc7e5-26e1-4fe9-ae46-e8311819e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931"
Oct 30 19:45:03.285766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.285742    5085 generic.go:356] "Generic (PLEG): container finished" podID="753847fd-164e-4b4c-bd22-7ac53910d283" containerID="b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5" exitCode=60
Oct 30 19:45:03.285870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.285855    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerDied","Data":"b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5"}
Oct 30 19:45:03.286179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.286154    5085 scope.go:122] "RemoveContainer" containerID="b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5"
Oct 30 19:45:03.286428 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:03.286406    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps_odh-ai-gateway-infra(753847fd-164e-4b4c-bd22-7ac53910d283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" podUID="753847fd-164e-4b4c-bd22-7ac53910d283"
Oct 30 19:45:03.387998 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.387975    5085 scope.go:122] "RemoveContainer" containerID="e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"
Oct 30 19:45:03.401799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.401777    5085 scope.go:122] "RemoveContainer" containerID="e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"
Oct 30 19:45:03.402114 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:03.402093    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3\": container with ID starting with e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3 not found: ID does not exist" containerID="e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"
Oct 30 19:45:03.402160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.402124    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3"} err="failed to get container status \"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3\": rpc error: code = NotFound desc = could not find container \"e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3\": container with ID starting with e046663f3250c59318eb69eb77ade512870b7cf18832c510c7170e32bfdb00e3 not found: ID does not exist"
Oct 30 19:45:03.402160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:03.402144    5085 scope.go:122] "RemoveContainer" containerID="585cb1ee4bc48c0b2ea9b512b2fcb578fee2bea503e20d1312b72b8428170cf8"
Oct 30 19:45:04.294770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.294738    5085 scope.go:122] "RemoveContainer" containerID="b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5"
Oct 30 19:45:04.295234 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:04.295063    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps_odh-ai-gateway-infra(753847fd-164e-4b4c-bd22-7ac53910d283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" podUID="753847fd-164e-4b4c-bd22-7ac53910d283"
Oct 30 19:45:04.296442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.296416    5085 scope.go:122] "RemoveContainer" containerID="3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431"
Oct 30 19:45:04.296722 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:04.296700    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7_odh-ai-gateway-infra(86db2ef4-1a34-4ca1-93d8-2ce8640f1a77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77"
Oct 30 19:45:04.298015 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.297980    5085 generic.go:356] "Generic (PLEG): container finished" podID="1988c7e9-48f2-466d-a343-3140593776f5" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f" exitCode=60
Oct 30 19:45:04.298113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.298054    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerDied","Data":"73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"}
Oct 30 19:45:04.298164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.298114    5085 scope.go:122] "RemoveContainer" containerID="fc18bdc7fda7dffaddc08a36dd649c159d892ac3bb4133e386d4b598eee3d40c"
Oct 30 19:45:04.298358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.298336    5085 scope.go:122] "RemoveContainer" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"
Oct 30 19:45:04.298619 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:04.298599    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n_odh-ai-gateway-infra(1988c7e9-48f2-466d-a343-3140593776f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" podUID="1988c7e9-48f2-466d-a343-3140593776f5"
Oct 30 19:45:04.300014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:04.300000    5085 scope.go:122] "RemoveContainer" containerID="db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71"
Oct 30 19:45:04.300205 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:04.300180    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-w4m8q_odh-ai-gateway-infra(611bc7e5-26e1-4fe9-ae46-e8311819e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931"
Oct 30 19:45:05.309428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:05.309402    5085 scope.go:122] "RemoveContainer" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"
Oct 30 19:45:05.309937 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:05.309726    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n_odh-ai-gateway-infra(1988c7e9-48f2-466d-a343-3140593776f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" podUID="1988c7e9-48f2-466d-a343-3140593776f5"
Oct 30 19:45:13.623968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:13.623935    5085 scope.go:122] "RemoveContainer" containerID="db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71"
Oct 30 19:45:14.373837 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:14.373752    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerStarted","Data":"6eee4b86a203fe2324cb0232b9c67eeb774b6fe211b42643da7c3328deaa9b54"}
Oct 30 19:45:14.487571 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:14.487504    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" podStartSLOduration=14.487490758 podStartE2EDuration="14.487490758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:14.483987079 +0000 UTC m=+1223.988390825" watchObservedRunningTime="2026-10-30 19:45:14.487490758 +0000 UTC m=+1223.991894532"
Oct 30 19:45:15.382519 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.382480    5085 generic.go:356] "Generic (PLEG): container finished" podID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerID="6eee4b86a203fe2324cb0232b9c67eeb774b6fe211b42643da7c3328deaa9b54" exitCode=60
Oct 30 19:45:15.382995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.382555    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerDied","Data":"6eee4b86a203fe2324cb0232b9c67eeb774b6fe211b42643da7c3328deaa9b54"}
Oct 30 19:45:15.382995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.382663    5085 scope.go:122] "RemoveContainer" containerID="db24754803a79af6b375762518fabc2bb86518c9d14dc5634fa22f01f70b9e71"
Oct 30 19:45:15.383094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.383039    5085 scope.go:122] "RemoveContainer" containerID="6eee4b86a203fe2324cb0232b9c67eeb774b6fe211b42643da7c3328deaa9b54"
Oct 30 19:45:15.383391 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:15.383362    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889825-w4m8q_odh-ai-gateway-infra(611bc7e5-26e1-4fe9-ae46-e8311819e931)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931"
Oct 30 19:45:15.511402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.511361    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:15.624768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:15.624738    5085 scope.go:122] "RemoveContainer" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"
Oct 30 19:45:16.391312 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.391231    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerStarted","Data":"a112cc38a2574b5e4393cde31764bff24666b697becf0492ab6fe5762dc81c00"}
Oct 30 19:45:16.502314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.502262    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" podStartSLOduration=16.502247497 podStartE2EDuration="16.502247497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:16.501289808 +0000 UTC m=+1226.005693554" watchObservedRunningTime="2026-10-30 19:45:16.502247497 +0000 UTC m=+1226.006651244"
Oct 30 19:45:16.552326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.552299    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:16.623982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.623945    5085 scope.go:122] "RemoveContainer" containerID="b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5"
Oct 30 19:45:16.624134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.623999    5085 scope.go:122] "RemoveContainer" containerID="3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431"
Oct 30 19:45:16.661475 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.661443    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\") pod \"611bc7e5-26e1-4fe9-ae46-e8311819e931\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") "
Oct 30 19:45:16.661662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.661614    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\" (UniqueName: \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\") pod \"611bc7e5-26e1-4fe9-ae46-e8311819e931\" (UID: \"611bc7e5-26e1-4fe9-ae46-e8311819e931\") "
Oct 30 19:45:16.661847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.661822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca" pod "611bc7e5-26e1-4fe9-ae46-e8311819e931" (UID: "611bc7e5-26e1-4fe9-ae46-e8311819e931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:16.661989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.661973    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/611bc7e5-26e1-4fe9-ae46-e8311819e931-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:16.663601 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.663553    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w" pod "611bc7e5-26e1-4fe9-ae46-e8311819e931" (UID: "611bc7e5-26e1-4fe9-ae46-e8311819e931"). InnerVolumeSpecName "kube-api-access-stw9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:16.762919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:16.762878    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stw9w\" (UniqueName: \"kubernetes.io/projected/611bc7e5-26e1-4fe9-ae46-e8311819e931-kube-api-access-stw9w\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:17.401896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.401803    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerStarted","Data":"67d9aa5ecf7d07c49c18de809d955ce3b2458aa98a5b80ee4fe5dd44c5264849"}
Oct 30 19:45:17.403681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.403654    5085 generic.go:356] "Generic (PLEG): container finished" podID="1988c7e9-48f2-466d-a343-3140593776f5" containerID="a112cc38a2574b5e4393cde31764bff24666b697becf0492ab6fe5762dc81c00" exitCode=60
Oct 30 19:45:17.403815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.403720    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerDied","Data":"a112cc38a2574b5e4393cde31764bff24666b697becf0492ab6fe5762dc81c00"}
Oct 30 19:45:17.403815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.403777    5085 scope.go:122] "RemoveContainer" containerID="73b4ba43f6354e895d94710f8bf405e0f090c97f1cb6e410ca201c9bf607a64f"
Oct 30 19:45:17.404121 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.404104    5085 scope.go:122] "RemoveContainer" containerID="a112cc38a2574b5e4393cde31764bff24666b697becf0492ab6fe5762dc81c00"
Oct 30 19:45:17.404415 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:17.404396    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n_odh-ai-gateway-infra(1988c7e9-48f2-466d-a343-3140593776f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" podUID="1988c7e9-48f2-466d-a343-3140593776f5"
Oct 30 19:45:17.405270 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.405251    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"
Oct 30 19:45:17.405335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.405270    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q" event={"ID":"611bc7e5-26e1-4fe9-ae46-e8311819e931","Type":"ContainerDied","Data":"d63473e962af28df9e1b2973e669568dcbd71f0c1bacb1652bcb161db6a9a976"}
Oct 30 19:45:17.407333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.407310    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerStarted","Data":"c683e612176ed8a21d89a044e2ff3932390a164905a79c6f36fd2b5fbd8e9e21"}
Oct 30 19:45:17.416935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.416918    5085 scope.go:122] "RemoveContainer" containerID="6eee4b86a203fe2324cb0232b9c67eeb774b6fe211b42643da7c3328deaa9b54"
Oct 30 19:45:17.516732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.516703    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:17.518208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.518161    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" podStartSLOduration=17.518148368 podStartE2EDuration="17.518148368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:17.517466259 +0000 UTC m=+1227.021870005" watchObservedRunningTime="2026-10-30 19:45:17.518148368 +0000 UTC m=+1227.022552117"
Oct 30 19:45:17.631273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.631246    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:17.645405 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.645376    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889825-w4m8q"]
Oct 30 19:45:17.864986 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.864931    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" podStartSLOduration=17.864914204 podStartE2EDuration="17.864914204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:17.861197753 +0000 UTC m=+1227.365601501" watchObservedRunningTime="2026-10-30 19:45:17.864914204 +0000 UTC m=+1227.369317963"
Oct 30 19:45:17.986183 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986141    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:45:17.986802 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986783    5085 cpu_manager.go:414] "containerMap: removing container" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.986923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986803    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.986923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986830    5085 cpu_manager.go:414] "containerMap: removing container" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.986923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986837    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.986923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986852    5085 cpu_manager.go:414] "containerMap: removing container" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.986923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.986860    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.987171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.987075    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.987171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.987092    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.987171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.987104    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" containerName="cleanup"
Oct 30 19:45:17.990744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:17.990723    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.053499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.053393    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:45:18.077097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077046    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077189    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077232    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077255    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h697f\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077384    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077425    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077504    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077546    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.077775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.077626    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178264 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178232    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178287    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178327    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178369    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178403    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178456    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178475    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178502    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h697f\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178849    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.178961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.178601    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.179173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.179150    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.179262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.179170    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.179311 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.179265    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.179369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.179342    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.180964 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.180934    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.181076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.181026    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.239186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.239154    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.239363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.239324    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h697f\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.276380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.276333    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:45:18.304166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.304131    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:18.544131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.540278    5085 generic.go:356] "Generic (PLEG): container finished" podID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerID="67d9aa5ecf7d07c49c18de809d955ce3b2458aa98a5b80ee4fe5dd44c5264849" exitCode=60
Oct 30 19:45:18.544131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.540439    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerDied","Data":"67d9aa5ecf7d07c49c18de809d955ce3b2458aa98a5b80ee4fe5dd44c5264849"}
Oct 30 19:45:18.544131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.540541    5085 scope.go:122] "RemoveContainer" containerID="3928dc74e4c7e69bb89f7c9568e2392e90fc685fb8d79ed263dd611278fab431"
Oct 30 19:45:18.544131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.541118    5085 scope.go:122] "RemoveContainer" containerID="67d9aa5ecf7d07c49c18de809d955ce3b2458aa98a5b80ee4fe5dd44c5264849"
Oct 30 19:45:18.544131 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:18.541637    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7_odh-ai-gateway-infra(86db2ef4-1a34-4ca1-93d8-2ce8640f1a77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77"
Oct 30 19:45:18.565823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.565787    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:18.580692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.580652    5085 generic.go:356] "Generic (PLEG): container finished" podID="753847fd-164e-4b4c-bd22-7ac53910d283" containerID="c683e612176ed8a21d89a044e2ff3932390a164905a79c6f36fd2b5fbd8e9e21" exitCode=60
Oct 30 19:45:18.580692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.580755    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerDied","Data":"c683e612176ed8a21d89a044e2ff3932390a164905a79c6f36fd2b5fbd8e9e21"}
Oct 30 19:45:18.581274 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.581250    5085 scope.go:122] "RemoveContainer" containerID="c683e612176ed8a21d89a044e2ff3932390a164905a79c6f36fd2b5fbd8e9e21"
Oct 30 19:45:18.581828 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:18.581644    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps_odh-ai-gateway-infra(753847fd-164e-4b4c-bd22-7ac53910d283)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" podUID="753847fd-164e-4b4c-bd22-7ac53910d283"
Oct 30 19:45:18.588879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.588816    5085 scope.go:122] "RemoveContainer" containerID="b2b67fabcc8a37ca8ae68dfb9ab7ce28909122b00b0668476acda7a3c936d2b5"
Oct 30 19:45:18.645379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.645252    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611bc7e5-26e1-4fe9-ae46-e8311819e931" path="/var/lib/kubelet/pods/611bc7e5-26e1-4fe9-ae46-e8311819e931/volumes"
Oct 30 19:45:18.713118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.713094    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:18.747642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.747613    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:45:18.748887 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:18.748861    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a50c7a_2ede_47ad_a22c_56796e8d3294.slice/crio-5ba5529db50bf8d20244b37714733b1b89081e508382a362ee061cf1e3200297 WatchSource:0}: Error finding container 5ba5529db50bf8d20244b37714733b1b89081e508382a362ee061cf1e3200297: Status 404 returned error can't find the container with id 5ba5529db50bf8d20244b37714733b1b89081e508382a362ee061cf1e3200297
Oct 30 19:45:18.751782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.751740    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:18.751860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.751836    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:18.751913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.751886    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:18.807125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.807042    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\" (UniqueName: \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\") pod \"1988c7e9-48f2-466d-a343-3140593776f5\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") "
Oct 30 19:45:18.807125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.807100    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\") pod \"1988c7e9-48f2-466d-a343-3140593776f5\" (UID: \"1988c7e9-48f2-466d-a343-3140593776f5\") "
Oct 30 19:45:18.807506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.807478    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca" pod "1988c7e9-48f2-466d-a343-3140593776f5" (UID: "1988c7e9-48f2-466d-a343-3140593776f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.809161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.809136    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b" pod "1988c7e9-48f2-466d-a343-3140593776f5" (UID: "1988c7e9-48f2-466d-a343-3140593776f5"). InnerVolumeSpecName "kube-api-access-rcj2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.881085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.881049    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:18.908673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.908642    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcj2b\" (UniqueName: \"kubernetes.io/projected/1988c7e9-48f2-466d-a343-3140593776f5-kube-api-access-rcj2b\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.908673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:18.908672    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1988c7e9-48f2-466d-a343-3140593776f5-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.608735 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.608425    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"
Oct 30 19:45:19.609332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.608418    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n" event={"ID":"1988c7e9-48f2-466d-a343-3140593776f5","Type":"ContainerDied","Data":"353b3920e5af37101cbf9701cf1466152e50c10aec0bc685b086e23c41ba32c9"}
Oct 30 19:45:19.609501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.609488    5085 scope.go:122] "RemoveContainer" containerID="a112cc38a2574b5e4393cde31764bff24666b697becf0492ab6fe5762dc81c00"
Oct 30 19:45:19.612304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.612276    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" event={"ID":"89a50c7a-2ede-47ad-a22c-56796e8d3294","Type":"ContainerStarted","Data":"0aec34dc0f8a294d62d713822a5363d50d8a0b498ec63bbb821f886d8c80c498"}
Oct 30 19:45:19.612549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.612523    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" event={"ID":"89a50c7a-2ede-47ad-a22c-56796e8d3294","Type":"ContainerStarted","Data":"5ba5529db50bf8d20244b37714733b1b89081e508382a362ee061cf1e3200297"}
Oct 30 19:45:19.897680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.897255    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:19.898753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.898726    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:19.902104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.902073    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bbfa07-29889825-xnk5n"]
Oct 30 19:45:19.923143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.923114    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\" (UniqueName: \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\") pod \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") "
Oct 30 19:45:19.923313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.923233    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\") pod \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\" (UID: \"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77\") "
Oct 30 19:45:19.923689 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.923661    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca" pod "86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" (UID: "86db2ef4-1a34-4ca1-93d8-2ce8640f1a77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:19.923820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.923804    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:19.925509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.925487    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc" pod "86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" (UID: "86db2ef4-1a34-4ca1-93d8-2ce8640f1a77"). InnerVolumeSpecName "kube-api-access-tq9tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:19.926241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:19.926224    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:20.025090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.025058    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\") pod \"753847fd-164e-4b4c-bd22-7ac53910d283\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") "
Oct 30 19:45:20.025305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.025136    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\" (UniqueName: \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\") pod \"753847fd-164e-4b4c-bd22-7ac53910d283\" (UID: \"753847fd-164e-4b4c-bd22-7ac53910d283\") "
Oct 30 19:45:20.025432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.025409    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca" pod "753847fd-164e-4b4c-bd22-7ac53910d283" (UID: "753847fd-164e-4b4c-bd22-7ac53910d283"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:20.025669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.025630    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9tc\" (UniqueName: \"kubernetes.io/projected/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77-kube-api-access-tq9tc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.025669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.025647    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/753847fd-164e-4b4c-bd22-7ac53910d283-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.027142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.027114    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr" pod "753847fd-164e-4b4c-bd22-7ac53910d283" (UID: "753847fd-164e-4b4c-bd22-7ac53910d283"). InnerVolumeSpecName "kube-api-access-zdpgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:20.088471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.088406    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podStartSLOduration=3.088390353 podStartE2EDuration="3.088390353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:20.084414386 +0000 UTC m=+1229.588818167" watchObservedRunningTime="2026-10-30 19:45:20.088390353 +0000 UTC m=+1229.592794100"
Oct 30 19:45:20.127083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.127046    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpgr\" (UniqueName: \"kubernetes.io/projected/753847fd-164e-4b4c-bd22-7ac53910d283-kube-api-access-zdpgr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:20.304367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.304316    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:20.305939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.305907    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:20.306446 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.305962    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:20.629674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.629592    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"
Oct 30 19:45:20.629674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.629600    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"
Oct 30 19:45:20.633906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.633876    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1988c7e9-48f2-466d-a343-3140593776f5" path="/var/lib/kubelet/pods/1988c7e9-48f2-466d-a343-3140593776f5/volumes"
Oct 30 19:45:20.634653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.634628    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps" event={"ID":"753847fd-164e-4b4c-bd22-7ac53910d283","Type":"ContainerDied","Data":"23bbfdc6fc6af73f6021950f2f5a753fe0cbb2811279eac6c355d687e4eafad5"}
Oct 30 19:45:20.634762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.634678    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7" event={"ID":"86db2ef4-1a34-4ca1-93d8-2ce8640f1a77","Type":"ContainerDied","Data":"299b855389411d2e4b11e0e4945db9ee0efd665601b2bbeb6dd5f524f0315173"}
Oct 30 19:45:20.634762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.634717    5085 scope.go:122] "RemoveContainer" containerID="c683e612176ed8a21d89a044e2ff3932390a164905a79c6f36fd2b5fbd8e9e21"
Oct 30 19:45:20.647579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.647546    5085 scope.go:122] "RemoveContainer" containerID="67d9aa5ecf7d07c49c18de809d955ce3b2458aa98a5b80ee4fe5dd44c5264849"
Oct 30 19:45:20.799201 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.799160    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:20.804777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.804748    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889825-5xdc7"]
Oct 30 19:45:20.901811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.901726    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:20.907996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:20.907966    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889825-8bpps"]
Oct 30 19:45:21.304696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:21.304661    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:21.304859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:21.304713    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:21.939843 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:21.939811    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:45:22.305130 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:22.305054    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:22.305130 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:22.305104    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:22.631955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:22.631882    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" path="/var/lib/kubelet/pods/753847fd-164e-4b4c-bd22-7ac53910d283/volumes"
Oct 30 19:45:22.632329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:22.632315    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" path="/var/lib/kubelet/pods/86db2ef4-1a34-4ca1-93d8-2ce8640f1a77/volumes"
Oct 30 19:45:23.305326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:23.305282    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:23.305828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:23.305340    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:24.305348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:24.305313    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:24.305893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:24.305372    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:25.304682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:25.304637    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 30 19:45:25.304870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:25.304721    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 30 19:45:26.309276 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:26.309244    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:26.309757 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:26.309672    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:26.310305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:26.310285    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:45:47.354868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.354838    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:45:47.355390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355376    5085 cpu_manager.go:414] "containerMap: removing container" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355391    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355411    5085 cpu_manager.go:414] "containerMap: removing container" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.355434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355416    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.355434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355426    5085 cpu_manager.go:414] "containerMap: removing container" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355431    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355444    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355449    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355462    5085 cpu_manager.go:414] "containerMap: removing container" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355467    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355474    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355478    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355492    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355497    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355602    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355613    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355618    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355627    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.355627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355632    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:45:47.356056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355638    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.356056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355646    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:45:47.356056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.355651    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:45:47.359553 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.359534    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.421423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.421389    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:45:47.452457 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452425    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452457 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452463    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452554    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdwd\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452642    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452683    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452714    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452737    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452767    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.452857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.452841    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554159    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554202    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554253    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdwd\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554322    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554358    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554478    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554648    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554661    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554696    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554778    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554780    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.554858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.554807    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.555084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.555060    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.555311 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.555292    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.556725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.556697    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.556887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.556869    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.615077 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.615006    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.615077 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.615066    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdwd\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:47.673319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:47.673276    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:48.097399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.097364    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:45:48.098172 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:45:48.098142    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd90aeaf_df2d_4f71_8250_2989bebbe1bc.slice/crio-ad454d5deef43182ac96e6d1527c5a508af76666de89367903837b45a825bb38 WatchSource:0}: Error finding container ad454d5deef43182ac96e6d1527c5a508af76666de89367903837b45a825bb38: Status 404 returned error can't find the container with id ad454d5deef43182ac96e6d1527c5a508af76666de89367903837b45a825bb38
Oct 30 19:45:48.100486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.100450    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:48.100596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.100513    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:48.100596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.100541    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:45:48.895050 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.895008    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" event={"ID":"dd90aeaf-df2d-4f71-8250-2989bebbe1bc","Type":"ContainerStarted","Data":"22a6ca1a129720fe63b8bee950bf0e4897534002d7da8f1a3cb54a2add476fd5"}
Oct 30 19:45:48.895454 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:48.895071    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" event={"ID":"dd90aeaf-df2d-4f71-8250-2989bebbe1bc","Type":"ContainerStarted","Data":"ad454d5deef43182ac96e6d1527c5a508af76666de89367903837b45a825bb38"}
Oct 30 19:45:49.108672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:49.108614    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podStartSLOduration=2.108597652 podStartE2EDuration="2.108597652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:49.057776477 +0000 UTC m=+1258.562180224" watchObservedRunningTime="2026-10-30 19:45:49.108597652 +0000 UTC m=+1258.613001398"
Oct 30 19:45:49.679412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:49.679376    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:49.681801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:49.681768    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:49.681953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:49.681825    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:50.677598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.677016    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:50.677598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.677079    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:50.882480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.882440    5085 scope.go:122] "RemoveContainer" containerID="0850f79716161660b46984e95ce326c3875badf29f93b8f3f7c7c66312a62a9d"
Oct 30 19:45:50.893001 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.892974    5085 scope.go:122] "RemoveContainer" containerID="0b3074408e32048271da551ed566cf0d04c8ac8e093ab2c0c423fc80201bcd75"
Oct 30 19:45:50.902146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.902127    5085 scope.go:122] "RemoveContainer" containerID="508a15966ab7389e84756b87aaf653e7ad46d8c011cf03026512b1a3253d69c9"
Oct 30 19:45:50.912953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:50.912931    5085 scope.go:122] "RemoveContainer" containerID="4f269dc888c5f9b16c2b03da41d54a94bed122b795ea67d07795f65bf0b62a96"
Oct 30 19:45:51.677587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:51.676936    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:51.677587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:51.676996    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:51.944547 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:45:51.944449    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:45:52.679588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:52.679015    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:52.679588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:52.679095    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:53.681711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:53.681669    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:53.682229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:53.681736    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:54.673850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:54.673813    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 30 19:45:54.674024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:54.673877    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 30 19:45:55.678629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:55.678600    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:55.681053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:55.678998    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:55.681053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:55.679722    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:45:58.718112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:45:58.718040    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:46:21.938869 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:46:21.938829    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:46:32.334654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.334574    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:46:32.701105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701063    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:46:32.701777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701755    5085 cpu_manager.go:414] "containerMap: removing container" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:46:32.701777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701777    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86db2ef4-1a34-4ca1-93d8-2ce8640f1a77" containerName="cleanup"
Oct 30 19:46:32.701969 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701813    5085 cpu_manager.go:414] "containerMap: removing container" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:46:32.701969 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701820    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="753847fd-164e-4b4c-bd22-7ac53910d283" containerName="cleanup"
Oct 30 19:46:32.702065 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.701994    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1988c7e9-48f2-466d-a343-3140593776f5" containerName="cleanup"
Oct 30 19:46:32.706083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.706057    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.807748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.807711    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:46:32.840511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.840472    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.842630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.842543    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.842923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.842892    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.843144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.843126    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.843348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.843332    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.843541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.843524    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.843742 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.843722    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.843919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.843903    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dkr\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.844112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.844095    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945379    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945515    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945549    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945630    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945679    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945747    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945813    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945852    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.945898    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dkr\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.946018    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.946607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.946396    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.947345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.946997    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.947731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.947702    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.947995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.947972    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.951305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.951246    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:32.962584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:32.958828    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:33.005262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.005210    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dkr\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:33.007431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.007375    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:33.029951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.029909    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:33.529996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.520538    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:46:33.543485 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:46:33.543434    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8604b620_8f84_40f0_8efc_229f8a352ec7.slice/crio-6a7d46a0117a90218964afe1d0c3bfab34a68fbd12942ac97ba836b0174ceae7 WatchSource:0}: Error finding container 6a7d46a0117a90218964afe1d0c3bfab34a68fbd12942ac97ba836b0174ceae7: Status 404 returned error can't find the container with id 6a7d46a0117a90218964afe1d0c3bfab34a68fbd12942ac97ba836b0174ceae7
Oct 30 19:46:33.571242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.570381    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:33.571242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.570481    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:33.571242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:33.570530    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:34.334856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:34.333863    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" event={"ID":"8604b620-8f84-40f0-8efc-229f8a352ec7","Type":"ContainerStarted","Data":"5e54c3647d07f9af180976f25e322233fd3a636cc6a6ab78d4b1c9f1986e0052"}
Oct 30 19:46:34.334856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:34.333932    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" event={"ID":"8604b620-8f84-40f0-8efc-229f8a352ec7","Type":"ContainerStarted","Data":"6a7d46a0117a90218964afe1d0c3bfab34a68fbd12942ac97ba836b0174ceae7"}
Oct 30 19:46:34.529214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:34.529152    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podStartSLOduration=2.529136237 podStartE2EDuration="2.529136237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:34.496685747 +0000 UTC m=+1304.001089496" watchObservedRunningTime="2026-10-30 19:46:34.529136237 +0000 UTC m=+1304.033539984"
Oct 30 19:46:35.031095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.031052    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:35.032506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.032478    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:35.032582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.032538    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:35.191467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.191429    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:46:35.195965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.195942    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.244904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.244871    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245064 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.244942    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245045    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245135    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245299 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245277    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245372    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245450    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245487    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.245633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.245606    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxl4\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.288636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.288062    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:46:35.347183 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347150    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347234    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347263    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347284    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347306    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347489    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxl4\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347592    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347612    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347652    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347672    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347708    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.347806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347757    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.348010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.347981    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.348140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.348113    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.349682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.349655    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.349843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.349827    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.405969 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.405938    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.406113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.406076    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxl4\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.510920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.510882    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:35.985117 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.985078    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:46:35.985224 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:46:35.985089    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b38ca1_bbdc_4cb3_b849_486bf3cca50b.slice/crio-dfddd22cf45ce5a01243738950ec2db37fb35a28a0230c3988d612a8c3b04952 WatchSource:0}: Error finding container dfddd22cf45ce5a01243738950ec2db37fb35a28a0230c3988d612a8c3b04952: Status 404 returned error can't find the container with id dfddd22cf45ce5a01243738950ec2db37fb35a28a0230c3988d612a8c3b04952
Oct 30 19:46:35.987865 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.987833    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:35.987939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.987899    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:35.987939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:35.987926    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:46:36.030663 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.030638    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:36.030779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.030687    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:36.349419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.349322    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" event={"ID":"73b38ca1-bbdc-4cb3-b849-486bf3cca50b","Type":"ContainerStarted","Data":"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"}
Oct 30 19:46:36.349419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.349383    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" event={"ID":"73b38ca1-bbdc-4cb3-b849-486bf3cca50b","Type":"ContainerStarted","Data":"dfddd22cf45ce5a01243738950ec2db37fb35a28a0230c3988d612a8c3b04952"}
Oct 30 19:46:36.474589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.471396    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podStartSLOduration=1.471375091 podStartE2EDuration="1.471375091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:36.470864972 +0000 UTC m=+1305.975268720" watchObservedRunningTime="2026-10-30 19:46:36.471375091 +0000 UTC m=+1305.975778838"
Oct 30 19:46:36.511023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:36.510985    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:37.031456 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:37.031414    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:37.031706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:37.031471    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:37.513482 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:37.513443    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:37.513884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:37.513520    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:38.031098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:38.031060    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:38.031287 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:38.031120    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:38.511626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:38.511314    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:38.511626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:38.511375    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:39.031792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:39.031756    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:39.032196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:39.031809    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:39.511467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:39.511414    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:39.511681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:39.511474    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:40.030939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:40.030894    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:46:40.031124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:40.030950    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 19:46:40.511584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:40.511540    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:40.511993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:40.511619    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:41.498618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:41.498585    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:41.499983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:41.499548    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:41.500278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:41.500258    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:46:41.512812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:41.512786    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:41.513263 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:41.513236    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:42.512031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:42.511990    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:46:42.512212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:42.512050    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 30 19:46:43.516711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:43.516681    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:43.517114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:43.517096    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:43.517720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:43.517703    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:46:50.136765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.136727    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"]
Oct 30 19:46:50.146138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.145177    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.254589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.232480    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"]
Oct 30 19:46:50.443648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.316688    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:50.443648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.316843    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:46:50.443648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.403213    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.443648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.404173    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfnh\" (UniqueName: \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.443648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.404343    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.707707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.707606    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.715911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.715875    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.716549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.715451    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.716855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.716835    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfnh\" (UniqueName: \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.767240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.766784    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:46:50.798592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.791353    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfnh\" (UniqueName: \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:50.840104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:50.840067    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6856691-vtq7r\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:51.243051 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:51.243011    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:46:51.266471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:51.251789    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:52.177676 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:46:52.162315    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:46:52.489415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:52.489392    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"]
Oct 30 19:46:52.490852 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:46:52.490821    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33315cd_a673_40be_b2ee_4fcb23ebcde8.slice/crio-5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7 WatchSource:0}: Error finding container 5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7: Status 404 returned error can't find the container with id 5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7
Oct 30 19:46:52.663946 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:52.663909    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r" event={"ID":"e33315cd-a673-40be-b2ee-4fcb23ebcde8","Type":"ContainerStarted","Data":"5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7"}
Oct 30 19:46:53.672263 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:53.672235    5085 generic.go:356] "Generic (PLEG): container finished" podID="e33315cd-a673-40be-b2ee-4fcb23ebcde8" containerID="d704ef2067760f605ca8ea6afe3779cb073f39434cef84b580c98b2e05e212c0" exitCode=0
Oct 30 19:46:53.672668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:53.672280    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r" event={"ID":"e33315cd-a673-40be-b2ee-4fcb23ebcde8","Type":"ContainerDied","Data":"d704ef2067760f605ca8ea6afe3779cb073f39434cef84b580c98b2e05e212c0"}
Oct 30 19:46:54.847252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.847225    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:54.881985 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.881959    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\") pod \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") "
Oct 30 19:46:54.882112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.882015    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\") pod \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") "
Oct 30 19:46:54.882195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.882182    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\" (UniqueName: \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\") pod \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\" (UID: \"e33315cd-a673-40be-b2ee-4fcb23ebcde8\") "
Oct 30 19:46:54.882339 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.882312    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca" pod "e33315cd-a673-40be-b2ee-4fcb23ebcde8" (UID: "e33315cd-a673-40be-b2ee-4fcb23ebcde8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:54.882617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.882590    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:54.884060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.884037    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca" pod "e33315cd-a673-40be-b2ee-4fcb23ebcde8" (UID: "e33315cd-a673-40be-b2ee-4fcb23ebcde8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:54.884146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.884108    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh" pod "e33315cd-a673-40be-b2ee-4fcb23ebcde8" (UID: "e33315cd-a673-40be-b2ee-4fcb23ebcde8"). InnerVolumeSpecName "kube-api-access-8lfnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:54.984284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.984209    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfnh\" (UniqueName: \"kubernetes.io/projected/e33315cd-a673-40be-b2ee-4fcb23ebcde8-kube-api-access-8lfnh\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:54.984284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:54.984237    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e33315cd-a673-40be-b2ee-4fcb23ebcde8-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:46:55.687008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:55.686978    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"
Oct 30 19:46:55.687008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:55.687001    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r" event={"ID":"e33315cd-a673-40be-b2ee-4fcb23ebcde8","Type":"ContainerDied","Data":"5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7"}
Oct 30 19:46:55.687232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:46:55.687049    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5171f307433c1dcddddf39b3571bbb0eb2cfa4926b636bed8fed599a5e7bbed7"
Oct 30 19:47:07.706053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.706021    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"]
Oct 30 19:47:07.706634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.706620    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e33315cd-a673-40be-b2ee-4fcb23ebcde8" containerName="revoke-keys"
Oct 30 19:47:07.706634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.706634    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33315cd-a673-40be-b2ee-4fcb23ebcde8" containerName="revoke-keys"
Oct 30 19:47:07.706764 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.706755    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33315cd-a673-40be-b2ee-4fcb23ebcde8" containerName="revoke-keys"
Oct 30 19:47:07.710303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.710283    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.765319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.765290    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"]
Oct 30 19:47:07.806780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.806747    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:07.806959 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.806839    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:07.835855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.835821    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.836021 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.835930    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54s96\" (UniqueName: \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.836064 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.836026    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.937738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.937701    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.937917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.937783    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54s96\" (UniqueName: \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.937917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.937855    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.938438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.938417    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.939958 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.939929    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:07.996537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:07.996466    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54s96\" (UniqueName: \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\") pod \"maas-api-revoke-sub-d4bb406e-xtvkk\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:08.024337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:08.024308    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:08.404043 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:08.404014    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5d372a_c5bb_4cfc_9ed8_68c0d7db9977.slice/crio-1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba WatchSource:0}: Error finding container 1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba: Status 404 returned error can't find the container with id 1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba
Oct 30 19:47:08.404261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:08.404240    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"]
Oct 30 19:47:08.780518 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:08.780477    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk" event={"ID":"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977","Type":"ContainerStarted","Data":"8c04d2ec75c014652e1fc0a7e13d5116dd84dd1d4d8ae64a3e6381de48606d79"}
Oct 30 19:47:08.780983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:08.780545    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk" event={"ID":"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977","Type":"ContainerStarted","Data":"1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba"}
Oct 30 19:47:08.894447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:08.894394    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk" podStartSLOduration=1.894380157 podStartE2EDuration="1.894380157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.891921781 +0000 UTC m=+1338.396325542" watchObservedRunningTime="2026-10-30 19:47:08.894380157 +0000 UTC m=+1338.398783903"
Oct 30 19:47:09.215287 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.215252    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"]
Oct 30 19:47:09.219891 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.219857    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.276262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.276231    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"]
Oct 30 19:47:09.359885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.359155    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.359885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.359691    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxsfb\" (UniqueName: \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.360353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.359917    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.461066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.461031    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxsfb\" (UniqueName: \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.461269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.461086    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.461269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.461175    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.461866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.461846    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.463475 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.463453    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.520091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.520018    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxsfb\" (UniqueName: \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\") pod \"maas-api-revoke-sub-a874ae99-qql84\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.534739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.534710    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:09.789160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.789073    5085 generic.go:356] "Generic (PLEG): container finished" podID="5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" containerID="8c04d2ec75c014652e1fc0a7e13d5116dd84dd1d4d8ae64a3e6381de48606d79" exitCode=0
Oct 30 19:47:09.789610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.789156    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk" event={"ID":"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977","Type":"ContainerDied","Data":"8c04d2ec75c014652e1fc0a7e13d5116dd84dd1d4d8ae64a3e6381de48606d79"}
Oct 30 19:47:09.910652 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:09.910624    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2895eb80_d625_4bb9_b05f_89936ce19e28.slice/crio-921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178 WatchSource:0}: Error finding container 921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178: Status 404 returned error can't find the container with id 921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178
Oct 30 19:47:09.953078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:09.953050    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"]
Oct 30 19:47:10.397710 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:10.397611    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:47:10.799307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:10.799269    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84" event={"ID":"2895eb80-d625-4bb9-b05f-89936ce19e28","Type":"ContainerStarted","Data":"53882d9ed4d43edd10ea5cce8caef03a80e0917e175945cf5efb5a95afd4be25"}
Oct 30 19:47:10.799815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:10.799334    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84" event={"ID":"2895eb80-d625-4bb9-b05f-89936ce19e28","Type":"ContainerStarted","Data":"921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178"}
Oct 30 19:47:10.921756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:10.921692    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84" podStartSLOduration=1.921676551 podStartE2EDuration="1.921676551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:10.91869524 +0000 UTC m=+1340.423098986" watchObservedRunningTime="2026-10-30 19:47:10.921676551 +0000 UTC m=+1340.426080324"
Oct 30 19:47:11.006594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.003947    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:11.092645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.092212    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\") pod \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") "
Oct 30 19:47:11.092645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.092279    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\" (UniqueName: \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\") pod \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") "
Oct 30 19:47:11.092645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.092397    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\") pod \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\" (UID: \"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977\") "
Oct 30 19:47:11.093592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.093320    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca" pod "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" (UID: "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:11.096962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.095967    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca" pod "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" (UID: "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:11.295839 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.295785    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96" pod "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" (UID: "5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977"). InnerVolumeSpecName "kube-api-access-54s96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:11.384643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.377270    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.384643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.377304    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54s96\" (UniqueName: \"kubernetes.io/projected/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-kube-api-access-54s96\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.384643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.377321    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:11.962555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.962512    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk" event={"ID":"5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977","Type":"ContainerDied","Data":"1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba"}
Oct 30 19:47:11.963038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.962595    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1092fb2989cf9fa39f122c86d6a71947881a6ea0d94d2f116e537aacd92a65ba"
Oct 30 19:47:11.963038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.962671    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"
Oct 30 19:47:11.997842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.997363    5085 generic.go:356] "Generic (PLEG): container finished" podID="2895eb80-d625-4bb9-b05f-89936ce19e28" containerID="53882d9ed4d43edd10ea5cce8caef03a80e0917e175945cf5efb5a95afd4be25" exitCode=0
Oct 30 19:47:11.997842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:11.997411    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84" event={"ID":"2895eb80-d625-4bb9-b05f-89936ce19e28","Type":"ContainerDied","Data":"53882d9ed4d43edd10ea5cce8caef03a80e0917e175945cf5efb5a95afd4be25"}
Oct 30 19:47:12.431585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:12.425680    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:12.463714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:12.463663    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:12.962717 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:12.962662    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:13.285752 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.285645    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:13.429594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.427692    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:13.429594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.428605    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:47:13.505892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.505842    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:13.507067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.507046    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:47:13.977590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.965652    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:47:13.977590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.967360    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.978204    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:13.988868    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.012502    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\") pod \"2895eb80-d625-4bb9-b05f-89936ce19e28\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") "
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.012768    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\" (UniqueName: \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\") pod \"2895eb80-d625-4bb9-b05f-89936ce19e28\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") "
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.012960    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\") pod \"2895eb80-d625-4bb9-b05f-89936ce19e28\" (UID: \"2895eb80-d625-4bb9-b05f-89936ce19e28\") "
Oct 30 19:47:14.030273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.014362    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca" pod "2895eb80-d625-4bb9-b05f-89936ce19e28" (UID: "2895eb80-d625-4bb9-b05f-89936ce19e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:14.093656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.088115    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca" pod "2895eb80-d625-4bb9-b05f-89936ce19e28" (UID: "2895eb80-d625-4bb9-b05f-89936ce19e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:14.173589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.153349    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:14.173589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.153386    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2895eb80-d625-4bb9-b05f-89936ce19e28-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:14.206400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.206362    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84" event={"ID":"2895eb80-d625-4bb9-b05f-89936ce19e28","Type":"ContainerDied","Data":"921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178"}
Oct 30 19:47:14.206704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.206676    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="921ed50284d334dd7b195d81434e66a7a23c48b0dfa247b351ac93b21cabe178"
Oct 30 19:47:14.206924 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.206912    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"
Oct 30 19:47:14.297796 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.297748    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:14.324588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.299464    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:47:14.400803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.400714    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb" pod "2895eb80-d625-4bb9-b05f-89936ce19e28" (UID: "2895eb80-d625-4bb9-b05f-89936ce19e28"). InnerVolumeSpecName "kube-api-access-hxsfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:14.440205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.440168    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:47:14.441277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.441244    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:47:14.468404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.468371    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxsfb\" (UniqueName: \"kubernetes.io/projected/2895eb80-d625-4bb9-b05f-89936ce19e28-kube-api-access-hxsfb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:14.490216 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.489049    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:47:14.555586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.552745    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:14.792981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.792671    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:47:14.792981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.792737    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:14.796832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.796805    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:47:14.816648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.804735    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:47:14.816648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:14.804786    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:15.019811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:15.019694    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:47:15.020404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:15.020364    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:15.239150 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:15.239115    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:47:18.483392 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.483356    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"]
Oct 30 19:47:18.484035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484000    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2895eb80-d625-4bb9-b05f-89936ce19e28" containerName="revoke-keys"
Oct 30 19:47:18.484035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484016    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2895eb80-d625-4bb9-b05f-89936ce19e28" containerName="revoke-keys"
Oct 30 19:47:18.484035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484026    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" containerName="revoke-keys"
Oct 30 19:47:18.484035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484033    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" containerName="revoke-keys"
Oct 30 19:47:18.484221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484181    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2895eb80-d625-4bb9-b05f-89936ce19e28" containerName="revoke-keys"
Oct 30 19:47:18.484221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.484192    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" containerName="revoke-keys"
Oct 30 19:47:18.487904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.487887    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.544807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.544778    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"]
Oct 30 19:47:18.586799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.586763    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:18.586979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.586817    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:18.590509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.590480    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfjx\" (UniqueName: \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.590604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.590525    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.590740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.590722    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.691749 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.691709    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.691922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.691820    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfjx\" (UniqueName: \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.691922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.691847    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.692611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.692554    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.694135 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.694115    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.750253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.750180    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfjx\" (UniqueName: \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\") pod \"maas-api-revoke-sub-210f37af-prlwk\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:18.801363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:18.801331    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:19.175223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:19.175194    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"]
Oct 30 19:47:19.175669 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:19.175643    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc6bf61_a8c3_4ffc_bd56_0f341a5605a4.slice/crio-3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd WatchSource:0}: Error finding container 3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd: Status 404 returned error can't find the container with id 3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd
Oct 30 19:47:19.249749 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:19.249709    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk" event={"ID":"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4","Type":"ContainerStarted","Data":"3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd"}
Oct 30 19:47:20.258667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:20.258629    5085 generic.go:356] "Generic (PLEG): container finished" podID="fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" containerID="80aeec79c660c975b27cd0289c178ffa2fbccb9eddc7433afffdfddd3231bdad" exitCode=0
Oct 30 19:47:20.259035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:20.258718    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk" event={"ID":"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4","Type":"ContainerDied","Data":"80aeec79c660c975b27cd0289c178ffa2fbccb9eddc7433afffdfddd3231bdad"}
Oct 30 19:47:21.421381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.421353    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:21.525052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.524957    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\" (UniqueName: \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\") pod \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") "
Oct 30 19:47:21.525052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.525023    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\") pod \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") "
Oct 30 19:47:21.525304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.525086    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\") pod \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\" (UID: \"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4\") "
Oct 30 19:47:21.525769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.525736    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca" pod "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" (UID: "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:21.527723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.527696    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca" pod "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" (UID: "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:21.527844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.527785    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx" pod "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" (UID: "fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4"). InnerVolumeSpecName "kube-api-access-rxfjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:21.627080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.627043    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfjx\" (UniqueName: \"kubernetes.io/projected/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-kube-api-access-rxfjx\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:21.627080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.627076    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:21.627080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:21.627086    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:21.939607 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:47:21.939545    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:47:22.273819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:22.273738    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk" event={"ID":"fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4","Type":"ContainerDied","Data":"3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd"}
Oct 30 19:47:22.273819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:22.273766    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"
Oct 30 19:47:22.273819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:22.273790    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3878af9cecc51fb83be3b03bbeeaa281fa0d26148f128575c464d4eb491776bd"
Oct 30 19:47:26.687807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.687771    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"]
Oct 30 19:47:26.688633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.688608    5085 cpu_manager.go:414] "containerMap: removing container" podUID="fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" containerName="revoke-keys"
Oct 30 19:47:26.688633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.688631    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" containerName="revoke-keys"
Oct 30 19:47:26.688834 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.688820    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" containerName="revoke-keys"
Oct 30 19:47:26.692542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.692520    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.742193 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.742163    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:26.786433 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.786390    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.786672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.786642    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.786776 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.786685    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btl7d\" (UniqueName: \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.788780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.788757    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:26.795078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.795053    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"]
Oct 30 19:47:26.888009 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.887974    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.888009 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.888008    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btl7d\" (UniqueName: \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.888229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.888150    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.888723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.888703    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.890509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.890487    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:26.946686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:26.946614    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btl7d\" (UniqueName: \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\") pod \"maas-api-revoke-sub-0a6e78c9-5bgdn\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:27.005694 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:27.005662    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:27.382070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:27.382039    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"]
Oct 30 19:47:27.383521 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:27.383494    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdcc549_d170_42ea_9d0e_73c174d9956e.slice/crio-ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db WatchSource:0}: Error finding container ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db: Status 404 returned error can't find the container with id ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db
Oct 30 19:47:28.316759 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:28.316720    5085 generic.go:356] "Generic (PLEG): container finished" podID="dcdcc549-d170-42ea-9d0e-73c174d9956e" containerID="e4fbe9f817c8a7a8eaa43f4127fc1e669e546c66595491c7b15e5c9162912c37" exitCode=0
Oct 30 19:47:28.317170 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:28.316810    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn" event={"ID":"dcdcc549-d170-42ea-9d0e-73c174d9956e","Type":"ContainerDied","Data":"e4fbe9f817c8a7a8eaa43f4127fc1e669e546c66595491c7b15e5c9162912c37"}
Oct 30 19:47:28.317170 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:28.316881    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn" event={"ID":"dcdcc549-d170-42ea-9d0e-73c174d9956e","Type":"ContainerStarted","Data":"ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db"}
Oct 30 19:47:29.491599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.491553    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:29.521661 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.521632    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\") pod \"dcdcc549-d170-42ea-9d0e-73c174d9956e\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") "
Oct 30 19:47:29.521859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.521685    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\" (UniqueName: \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\") pod \"dcdcc549-d170-42ea-9d0e-73c174d9956e\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") "
Oct 30 19:47:29.521859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.521719    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\") pod \"dcdcc549-d170-42ea-9d0e-73c174d9956e\" (UID: \"dcdcc549-d170-42ea-9d0e-73c174d9956e\") "
Oct 30 19:47:29.522114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.522080    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca" pod "dcdcc549-d170-42ea-9d0e-73c174d9956e" (UID: "dcdcc549-d170-42ea-9d0e-73c174d9956e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:29.523877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.523853    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca" pod "dcdcc549-d170-42ea-9d0e-73c174d9956e" (UID: "dcdcc549-d170-42ea-9d0e-73c174d9956e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:29.523962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.523934    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d" pod "dcdcc549-d170-42ea-9d0e-73c174d9956e" (UID: "dcdcc549-d170-42ea-9d0e-73c174d9956e"). InnerVolumeSpecName "kube-api-access-btl7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:29.623625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.623505    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.623625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.623535    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btl7d\" (UniqueName: \"kubernetes.io/projected/dcdcc549-d170-42ea-9d0e-73c174d9956e-kube-api-access-btl7d\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:29.623625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:29.623547    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcdcc549-d170-42ea-9d0e-73c174d9956e-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:30.332491 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:30.332457    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn" event={"ID":"dcdcc549-d170-42ea-9d0e-73c174d9956e","Type":"ContainerDied","Data":"ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db"}
Oct 30 19:47:30.332680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:30.332515    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7f385b6c2778215fe565dde2b707b1f05b9882d33a6d2a7c5332b2752414db"
Oct 30 19:47:30.332680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:30.332465    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"
Oct 30 19:47:33.220692 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.220638    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:47:33.222181 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.222150    5085 cpu_manager.go:414] "containerMap: removing container" podUID="dcdcc549-d170-42ea-9d0e-73c174d9956e" containerName="revoke-keys"
Oct 30 19:47:33.222331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.222317    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdcc549-d170-42ea-9d0e-73c174d9956e" containerName="revoke-keys"
Oct 30 19:47:33.222679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.222662    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdcc549-d170-42ea-9d0e-73c174d9956e" containerName="revoke-keys"
Oct 30 19:47:33.349824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.349787    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.413588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.391448    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:47:33.456589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.453075    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:33.507643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.482925    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:33.548590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.503404    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.548979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.548943    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.549127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.549108    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.555796 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.555755    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.583586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.559699    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.620587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.585508    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.690611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.597388    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:47:33.690611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.607516    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:47:33.690611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.686378    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.690611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.686450    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qm7\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.690611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.686516    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.761723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.760881    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801377    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801443    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801470    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801507    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qm7\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801550    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801644    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801742    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801768    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.801885    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.802682    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.802973    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.819589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.819029    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.820364 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.820053    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.820364 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.820347    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.855944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.855904    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.888587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.880284    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.911585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.904342    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qm7\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.911585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.907160    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\") pod \"e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") " pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:33.921071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.919657    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:33.952402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:33.952313    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:34.108673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.108631    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"]
Oct 30 19:47:34.142737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.142696    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfeb379-ns879"]
Oct 30 19:47:34.453748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.453703    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:47:34.480668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.480631    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:47:34.523699 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.523654    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:34.524673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.524647    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:47:34.813697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.800044    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:47:34.813697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.800824    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:47:34.813697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.801584    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:47:34.813697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.801640    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:47:34.817604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.814141    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:47:34.817604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:34.814516    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:47:35.192114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.164816    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0e0a33-294b-4a98-baeb-65d299391c17" path="/var/lib/kubelet/pods/7a0e0a33-294b-4a98-baeb-65d299391c17/volumes"
Oct 30 19:47:35.347078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.336324    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:47:35.347078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.336427    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:47:35.379140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.369191    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:47:35.435828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.389615    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:47:35.435828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.410026    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:47:35.435828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.410181    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:47:35.435828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.410280    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:47:35.435828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.433385    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:47:35.462531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.462497    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:47:35.928536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.928483    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"]
Oct 30 19:47:35.933461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.933437    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:35.981338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.981297    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"]
Oct 30 19:47:35.983468 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.983442    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:35.983651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.983474    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:35.999644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.999616    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7js\" (UniqueName: \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:35.999842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.999818    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:35.999897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:35.999870    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.101912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.101873    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7js\" (UniqueName: \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.102087 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.102049    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.102177 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.102092    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.102970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.102946    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.104548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.104528    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.162394 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.162361    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7js\" (UniqueName: \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\") pod \"maas-api-revoke-sub-23bbe81b-gl7m5\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.251447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.251326    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:36.393589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.393065    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" event={"ID":"ebe38533-bc3c-4180-b45e-a10b909aef47","Type":"ContainerStarted","Data":"1d18810d7e9372c6366129103b52d8cba6302412e01533871ad92b433c8c2451"}
Oct 30 19:47:36.393589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.393175    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" event={"ID":"ebe38533-bc3c-4180-b45e-a10b909aef47","Type":"ContainerStarted","Data":"0272737e6baaaea6b7de777e4b5eed0ad579af1db4ee8d1479d3f09dd10e2ed4"}
Oct 30 19:47:36.518998 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.518904    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podStartSLOduration=3.518882874 podStartE2EDuration="3.518882874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:36.514521686 +0000 UTC m=+1366.018925433" watchObservedRunningTime="2026-10-30 19:47:36.518882874 +0000 UTC m=+1366.023286623"
Oct 30 19:47:36.631626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.631601    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"]
Oct 30 19:47:36.631870 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:36.631848    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e03397_4b6a_4aef_bdb7_c1d4eac90f4d.slice/crio-410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4 WatchSource:0}: Error finding container 410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4: Status 404 returned error can't find the container with id 410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4
Oct 30 19:47:36.952867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.952827    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:36.954202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.954174    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:36.954337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:36.954225    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:37.402975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.402932    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5" event={"ID":"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d","Type":"ContainerStarted","Data":"a2e256c1aef91e1f3bae598a50a973a8db29275aaa664832ca3942645eda0107"}
Oct 30 19:47:37.403172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.403000    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5" event={"ID":"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d","Type":"ContainerStarted","Data":"410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4"}
Oct 30 19:47:37.517069 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.517010    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5" podStartSLOduration=2.516994059 podStartE2EDuration="2.516994059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:37.514265214 +0000 UTC m=+1367.018668962" watchObservedRunningTime="2026-10-30 19:47:37.516994059 +0000 UTC m=+1367.021397850"
Oct 30 19:47:37.525667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.525636    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"]
Oct 30 19:47:37.530281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.530262    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.589339 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.589307    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"]
Oct 30 19:47:37.626638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.626597    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.626806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.626655    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.626806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.626735    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmrs\" (UniqueName: \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.728635 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.728588    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.728635 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.728663    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.729022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.728708    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmrs\" (UniqueName: \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.729353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.729325    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.731199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.731169    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.787655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.787624    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmrs\" (UniqueName: \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\") pod \"maas-api-revoke-sub-6e4f1b52-74zn5\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.844363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.844333    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:37.953163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.953124    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:37.953305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:37.953191    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:38.291004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.290961    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"]
Oct 30 19:47:38.412008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.411974    5085 generic.go:356] "Generic (PLEG): container finished" podID="a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" containerID="a2e256c1aef91e1f3bae598a50a973a8db29275aaa664832ca3942645eda0107" exitCode=0
Oct 30 19:47:38.412182 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.412041    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5" event={"ID":"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d","Type":"ContainerDied","Data":"a2e256c1aef91e1f3bae598a50a973a8db29275aaa664832ca3942645eda0107"}
Oct 30 19:47:38.413586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.413532    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5" event={"ID":"d3b4e163-fd71-4eaf-bbab-387c76879cc6","Type":"ContainerStarted","Data":"25a4716fb114c6c81feed026ed06aad627c3bfc44af9e8a15cbae78a4a764bcc"}
Oct 30 19:47:38.953914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.953831    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:38.953914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:38.953885    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:39.422296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.422265    5085 generic.go:356] "Generic (PLEG): container finished" podID="d3b4e163-fd71-4eaf-bbab-387c76879cc6" containerID="cc4bad6cae2840cceff6024f5dce3e25b970b38328db1a39be6933e43d3c46f4" exitCode=0
Oct 30 19:47:39.422482 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.422345    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5" event={"ID":"d3b4e163-fd71-4eaf-bbab-387c76879cc6","Type":"ContainerDied","Data":"cc4bad6cae2840cceff6024f5dce3e25b970b38328db1a39be6933e43d3c46f4"}
Oct 30 19:47:39.613373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.613345    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:39.655524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.655495    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\") pod \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") "
Oct 30 19:47:39.655731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.655697    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\" (UniqueName: \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\") pod \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") "
Oct 30 19:47:39.655783 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.655737    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\") pod \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\" (UID: \"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d\") "
Oct 30 19:47:39.656265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.656233    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca" pod "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" (UID: "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:39.683162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.683121    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js" pod "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" (UID: "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d"). InnerVolumeSpecName "kube-api-access-5g7js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:39.683162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.683133    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca" pod "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" (UID: "a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:39.782742 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.782704    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.782996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.782976    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g7js\" (UniqueName: \"kubernetes.io/projected/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-kube-api-access-5g7js\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.783107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.783091    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:39.953286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.953254    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:39.953462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:39.953305    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:40.432214 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.432186    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"
Oct 30 19:47:40.432656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.432218    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5" event={"ID":"a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d","Type":"ContainerDied","Data":"410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4"}
Oct 30 19:47:40.432656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.432281    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410f6035b28843b60950eb65ba66cb7e56224a71517f6c96f50b32d1c617cce4"
Oct 30 19:47:40.631256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.631227    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:40.694361 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.694277    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\") pod \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") "
Oct 30 19:47:40.694520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.694385    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\" (UniqueName: \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\") pod \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") "
Oct 30 19:47:40.694739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.694715    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\") pod \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\" (UID: \"d3b4e163-fd71-4eaf-bbab-387c76879cc6\") "
Oct 30 19:47:40.695052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.695024    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca" pod "d3b4e163-fd71-4eaf-bbab-387c76879cc6" (UID: "d3b4e163-fd71-4eaf-bbab-387c76879cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:40.695228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.695211    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:40.696280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.696257    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca" pod "d3b4e163-fd71-4eaf-bbab-387c76879cc6" (UID: "d3b4e163-fd71-4eaf-bbab-387c76879cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:40.696598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.696553    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs" pod "d3b4e163-fd71-4eaf-bbab-387c76879cc6" (UID: "d3b4e163-fd71-4eaf-bbab-387c76879cc6"). InnerVolumeSpecName "kube-api-access-jfmrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:40.796890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.796851    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b4e163-fd71-4eaf-bbab-387c76879cc6-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:40.796890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.796887    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmrs\" (UniqueName: \"kubernetes.io/projected/d3b4e163-fd71-4eaf-bbab-387c76879cc6-kube-api-access-jfmrs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:40.952831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.952747    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:40.952831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:40.952802    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:41.446526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:41.446493    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"
Oct 30 19:47:41.447004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:41.446576    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5" event={"ID":"d3b4e163-fd71-4eaf-bbab-387c76879cc6","Type":"ContainerDied","Data":"25a4716fb114c6c81feed026ed06aad627c3bfc44af9e8a15cbae78a4a764bcc"}
Oct 30 19:47:41.447004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:41.446624    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25a4716fb114c6c81feed026ed06aad627c3bfc44af9e8a15cbae78a4a764bcc"
Oct 30 19:47:41.953014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:41.952972    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 19:47:41.953235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:41.953038    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 19:47:42.481325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.481281    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"]
Oct 30 19:47:42.481973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.481953    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" containerName="revoke-keys"
Oct 30 19:47:42.481973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.481970    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" containerName="revoke-keys"
Oct 30 19:47:42.482165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.481982    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d3b4e163-fd71-4eaf-bbab-387c76879cc6" containerName="revoke-keys"
Oct 30 19:47:42.482165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.481987    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b4e163-fd71-4eaf-bbab-387c76879cc6" containerName="revoke-keys"
Oct 30 19:47:42.482165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.482123    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" containerName="revoke-keys"
Oct 30 19:47:42.482165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.482133    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b4e163-fd71-4eaf-bbab-387c76879cc6" containerName="revoke-keys"
Oct 30 19:47:42.485927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.485899    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.541442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.541403    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"]
Oct 30 19:47:42.582412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.582384    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:42.582762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.582707    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:42.629319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.629280    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.629485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.629461    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrsbj\" (UniqueName: \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.629597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.629518    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.730356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.730315    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrsbj\" (UniqueName: \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.730551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.730376    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.730551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.730510    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.731272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.731242    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.733443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.733389    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.789764 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.789726    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrsbj\" (UniqueName: \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\") pod \"maas-api-revoke-sub-674eceb5-l9j5s\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.804551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.804495    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:42.957283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.957256    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:42.957706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.957679    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:42.958334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:42.958315    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:47:43.184850 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:43.184821    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbb98ca_f0e6_4260_a2b9_c488c1159fd1.slice/crio-47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305 WatchSource:0}: Error finding container 47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305: Status 404 returned error can't find the container with id 47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305
Oct 30 19:47:43.185370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:43.185315    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"]
Oct 30 19:47:43.462238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:43.462206    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s" event={"ID":"abbb98ca-f0e6-4260-a2b9-c488c1159fd1","Type":"ContainerStarted","Data":"47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305"}
Oct 30 19:47:44.471232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:44.471194    5085 generic.go:356] "Generic (PLEG): container finished" podID="abbb98ca-f0e6-4260-a2b9-c488c1159fd1" containerID="411951bbfb6ebff5370cf0f4671fd0d36533e09b0520996cb71df1d882ee4b79" exitCode=0
Oct 30 19:47:44.471666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:44.471227    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s" event={"ID":"abbb98ca-f0e6-4260-a2b9-c488c1159fd1","Type":"ContainerDied","Data":"411951bbfb6ebff5370cf0f4671fd0d36533e09b0520996cb71df1d882ee4b79"}
Oct 30 19:47:45.652933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.652905    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:45.770097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.770019    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\") pod \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") "
Oct 30 19:47:45.770097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.770063    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\") pod \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") "
Oct 30 19:47:45.770344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.770192    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\" (UniqueName: \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\") pod \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\" (UID: \"abbb98ca-f0e6-4260-a2b9-c488c1159fd1\") "
Oct 30 19:47:45.770457 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.770430    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca" pod "abbb98ca-f0e6-4260-a2b9-c488c1159fd1" (UID: "abbb98ca-f0e6-4260-a2b9-c488c1159fd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:45.770857 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.770833    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:45.772554 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.772521    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj" pod "abbb98ca-f0e6-4260-a2b9-c488c1159fd1" (UID: "abbb98ca-f0e6-4260-a2b9-c488c1159fd1"). InnerVolumeSpecName "kube-api-access-nrsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:45.772676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.772615    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca" pod "abbb98ca-f0e6-4260-a2b9-c488c1159fd1" (UID: "abbb98ca-f0e6-4260-a2b9-c488c1159fd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:45.872473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.872430    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:45.872473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:45.872461    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrsbj\" (UniqueName: \"kubernetes.io/projected/abbb98ca-f0e6-4260-a2b9-c488c1159fd1-kube-api-access-nrsbj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:46.488252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:46.488224    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"
Oct 30 19:47:46.488451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:46.488263    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s" event={"ID":"abbb98ca-f0e6-4260-a2b9-c488c1159fd1","Type":"ContainerDied","Data":"47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305"}
Oct 30 19:47:46.488451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:46.488318    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b1fd0be3f36702344d5e5cc99149fcb650f6fa957ba2370fa952bad56f0305"
Oct 30 19:47:51.052662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:51.052629    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"]
Oct 30 19:47:51.099371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:51.099338    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8b269d-fck75"]
Oct 30 19:47:51.943515 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:47:51.943476    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:47:52.631598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:52.631546    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013" path="/var/lib/kubelet/pods/81bbb91a-dcd7-4ebd-9af8-bb2dd1a04013/volumes"
Oct 30 19:47:53.726451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.726417    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"]
Oct 30 19:47:53.727095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.727067    5085 cpu_manager.go:414] "containerMap: removing container" podUID="abbb98ca-f0e6-4260-a2b9-c488c1159fd1" containerName="revoke-keys"
Oct 30 19:47:53.727095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.727088    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbb98ca-f0e6-4260-a2b9-c488c1159fd1" containerName="revoke-keys"
Oct 30 19:47:53.727265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.727236    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbb98ca-f0e6-4260-a2b9-c488c1159fd1" containerName="revoke-keys"
Oct 30 19:47:53.731979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.731960    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.765411 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.765381    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.765584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.765439    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jwb\" (UniqueName: \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.765704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.765682    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.787479 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.787449    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"]
Oct 30 19:47:53.828646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.828605    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:47:53.828826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.828682    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:47:53.866879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.866844    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.867079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.866911    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.867079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.866953    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jwb\" (UniqueName: \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.867658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.867636    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.869350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.869328    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.926613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.926556    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jwb\" (UniqueName: \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\") pod \"maas-api-revoke-sub-7b1ad1e6-vs5zb\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:53.988736 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:53.988655    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:47:54.050983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:54.050953    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:54.457583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:54.456641    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"]
Oct 30 19:47:54.563794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:54.563755    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb" event={"ID":"d9e19118-1d38-4faa-9543-4e17c51cad2c","Type":"ContainerStarted","Data":"3647049b6ada8c68ae3b3d3d58982b6cfc10e9a26cc7e879b301cb5669f3209d"}
Oct 30 19:47:54.986336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:54.986295    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"]
Oct 30 19:47:54.993224 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:54.993191    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.049212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.049164    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"]
Oct 30 19:47:55.425448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.425413    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.430027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.429995    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.430225 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.430205    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qhr\" (UniqueName: \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.531612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.531493    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.533615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.533586    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.533794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.533776    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qhr\" (UniqueName: \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.534349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.532662    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.568441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.551004    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.617975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.617936    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb" event={"ID":"d9e19118-1d38-4faa-9543-4e17c51cad2c","Type":"ContainerStarted","Data":"0fd4741b39471dbc17b577086b511dbfd0e84cd0188046b3a64170dc474e2632"}
Oct 30 19:47:55.620292 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.620262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qhr\" (UniqueName: \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\") pod \"maas-api-revoke-sub-e4893bd8-nbcpl\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.645215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.645175    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:55.798647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:55.771280    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb" podStartSLOduration=2.771261633 podStartE2EDuration="2.771261633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:55.769545825 +0000 UTC m=+1385.273949575" watchObservedRunningTime="2026-10-30 19:47:55.771261633 +0000 UTC m=+1385.275665381"
Oct 30 19:47:56.666815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.664278    5085 generic.go:356] "Generic (PLEG): container finished" podID="d9e19118-1d38-4faa-9543-4e17c51cad2c" containerID="0fd4741b39471dbc17b577086b511dbfd0e84cd0188046b3a64170dc474e2632" exitCode=0
Oct 30 19:47:56.671099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.671068    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb" event={"ID":"d9e19118-1d38-4faa-9543-4e17c51cad2c","Type":"ContainerDied","Data":"0fd4741b39471dbc17b577086b511dbfd0e84cd0188046b3a64170dc474e2632"}
Oct 30 19:47:56.671300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.671283    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl" event={"ID":"9ce394d1-3507-4d6a-80ef-cfa36122fc07","Type":"ContainerStarted","Data":"1f4c6b53fec2f85413496699fe6518697ada04081e446f5190753e6e681fd79f"}
Oct 30 19:47:56.672435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.672412    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"]
Oct 30 19:47:56.685054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.685009    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"]
Oct 30 19:47:56.692843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.692819    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.739149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.739119    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"]
Oct 30 19:47:56.774841 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:47:56.774754    5085 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e19118_1d38_4faa_9543_4e17c51cad2c.slice/crio-conmon-0fd4741b39471dbc17b577086b511dbfd0e84cd0188046b3a64170dc474e2632.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 19:47:56.829138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.829076    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgkk\" (UniqueName: \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.829500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.829473    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.829783 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.829754    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.931631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.931592    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgkk\" (UniqueName: \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.931814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.931713    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.931814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.931798    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.932313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.932284    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.934090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.934070    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:56.992145 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:56.992112    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgkk\" (UniqueName: \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\") pod \"maas-api-revoke-sub-5174c56c-bj79z\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:57.070035 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.070002    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:57.463188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.463159    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"]
Oct 30 19:47:57.464314 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:47:57.464283    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1736694d_91fc_4895_8339_7754f5996c3a.slice/crio-8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9 WatchSource:0}: Error finding container 8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9: Status 404 returned error can't find the container with id 8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9
Oct 30 19:47:57.679218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.679185    5085 generic.go:356] "Generic (PLEG): container finished" podID="9ce394d1-3507-4d6a-80ef-cfa36122fc07" containerID="50ec058530413c2d9c5629923b1c484841f59aab0480d27f0bcb2ba0990ba1ac" exitCode=0
Oct 30 19:47:57.679633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.679277    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl" event={"ID":"9ce394d1-3507-4d6a-80ef-cfa36122fc07","Type":"ContainerDied","Data":"50ec058530413c2d9c5629923b1c484841f59aab0480d27f0bcb2ba0990ba1ac"}
Oct 30 19:47:57.680731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.680707    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z" event={"ID":"1736694d-91fc-4895-8339-7754f5996c3a","Type":"ContainerStarted","Data":"8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9"}
Oct 30 19:47:57.825300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.825273    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:57.949943 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.949912    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\" (UniqueName: \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\") pod \"d9e19118-1d38-4faa-9543-4e17c51cad2c\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") "
Oct 30 19:47:57.950103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.950087    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\") pod \"d9e19118-1d38-4faa-9543-4e17c51cad2c\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") "
Oct 30 19:47:57.950188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.950143    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\") pod \"d9e19118-1d38-4faa-9543-4e17c51cad2c\" (UID: \"d9e19118-1d38-4faa-9543-4e17c51cad2c\") "
Oct 30 19:47:57.950610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.950556    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca" pod "d9e19118-1d38-4faa-9543-4e17c51cad2c" (UID: "d9e19118-1d38-4faa-9543-4e17c51cad2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:57.950755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.950736    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:57.952178 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.952152    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb" pod "d9e19118-1d38-4faa-9543-4e17c51cad2c" (UID: "d9e19118-1d38-4faa-9543-4e17c51cad2c"). InnerVolumeSpecName "kube-api-access-47jwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:57.952910 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:57.952889    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca" pod "d9e19118-1d38-4faa-9543-4e17c51cad2c" (UID: "d9e19118-1d38-4faa-9543-4e17c51cad2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:58.052004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.051922    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jwb\" (UniqueName: \"kubernetes.io/projected/d9e19118-1d38-4faa-9543-4e17c51cad2c-kube-api-access-47jwb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.052004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.051952    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9e19118-1d38-4faa-9543-4e17c51cad2c-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.688890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.688815    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"
Oct 30 19:47:58.689275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.688814    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb" event={"ID":"d9e19118-1d38-4faa-9543-4e17c51cad2c","Type":"ContainerDied","Data":"3647049b6ada8c68ae3b3d3d58982b6cfc10e9a26cc7e879b301cb5669f3209d"}
Oct 30 19:47:58.689275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.688942    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3647049b6ada8c68ae3b3d3d58982b6cfc10e9a26cc7e879b301cb5669f3209d"
Oct 30 19:47:58.690425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.690400    5085 generic.go:356] "Generic (PLEG): container finished" podID="1736694d-91fc-4895-8339-7754f5996c3a" containerID="0c8a72d0560cc3e1dbca7a9c86b37c60fa974ec6a03a89221d1f8140c52cc377" exitCode=0
Oct 30 19:47:58.690576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.690443    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z" event={"ID":"1736694d-91fc-4895-8339-7754f5996c3a","Type":"ContainerDied","Data":"0c8a72d0560cc3e1dbca7a9c86b37c60fa974ec6a03a89221d1f8140c52cc377"}
Oct 30 19:47:58.854185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.854161    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:58.962472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.962390    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\" (UniqueName: \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\") pod \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") "
Oct 30 19:47:58.962640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.962478    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\") pod \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") "
Oct 30 19:47:58.962640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.962579    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\") pod \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\" (UID: \"9ce394d1-3507-4d6a-80ef-cfa36122fc07\") "
Oct 30 19:47:58.962858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.962832    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca" pod "9ce394d1-3507-4d6a-80ef-cfa36122fc07" (UID: "9ce394d1-3507-4d6a-80ef-cfa36122fc07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:58.963162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.963139    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:58.964645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.964619    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca" pod "9ce394d1-3507-4d6a-80ef-cfa36122fc07" (UID: "9ce394d1-3507-4d6a-80ef-cfa36122fc07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:47:58.964695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:58.964627    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr" pod "9ce394d1-3507-4d6a-80ef-cfa36122fc07" (UID: "9ce394d1-3507-4d6a-80ef-cfa36122fc07"). InnerVolumeSpecName "kube-api-access-78qhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:59.064696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.064653    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce394d1-3507-4d6a-80ef-cfa36122fc07-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.064696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.064688    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qhr\" (UniqueName: \"kubernetes.io/projected/9ce394d1-3507-4d6a-80ef-cfa36122fc07-kube-api-access-78qhr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:59.699417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.699378    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl" event={"ID":"9ce394d1-3507-4d6a-80ef-cfa36122fc07","Type":"ContainerDied","Data":"1f4c6b53fec2f85413496699fe6518697ada04081e446f5190753e6e681fd79f"}
Oct 30 19:47:59.699868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.699446    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4c6b53fec2f85413496699fe6518697ada04081e446f5190753e6e681fd79f"
Oct 30 19:47:59.699868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.699401    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"
Oct 30 19:47:59.845402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.845378    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:47:59.976842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.976752    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\" (UniqueName: \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\") pod \"1736694d-91fc-4895-8339-7754f5996c3a\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") "
Oct 30 19:47:59.977016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.976942    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\") pod \"1736694d-91fc-4895-8339-7754f5996c3a\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") "
Oct 30 19:47:59.977016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.976965    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\") pod \"1736694d-91fc-4895-8339-7754f5996c3a\" (UID: \"1736694d-91fc-4895-8339-7754f5996c3a\") "
Oct 30 19:47:59.977441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.977401    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca" pod "1736694d-91fc-4895-8339-7754f5996c3a" (UID: "1736694d-91fc-4895-8339-7754f5996c3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:47:59.979015 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.978987    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk" pod "1736694d-91fc-4895-8339-7754f5996c3a" (UID: "1736694d-91fc-4895-8339-7754f5996c3a"). InnerVolumeSpecName "kube-api-access-kmgkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:59.979679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:47:59.979659    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca" pod "1736694d-91fc-4895-8339-7754f5996c3a" (UID: "1736694d-91fc-4895-8339-7754f5996c3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:00.078385 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.078342    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:00.078385 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.078377    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1736694d-91fc-4895-8339-7754f5996c3a-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:00.078385 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.078391    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgkk\" (UniqueName: \"kubernetes.io/projected/1736694d-91fc-4895-8339-7754f5996c3a-kube-api-access-kmgkk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:00.708196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.708168    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"
Oct 30 19:48:00.708593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.708204    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z" event={"ID":"1736694d-91fc-4895-8339-7754f5996c3a","Type":"ContainerDied","Data":"8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9"}
Oct 30 19:48:00.708593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:00.708249    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea0e86a1916ceea47d6a321a55ce251861993f4cd67b07d918ce9d4635e89c9"
Oct 30 19:48:04.515782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.515709    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"]
Oct 30 19:48:04.516656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516635    5085 cpu_manager.go:414] "containerMap: removing container" podUID="9ce394d1-3507-4d6a-80ef-cfa36122fc07" containerName="revoke-keys"
Oct 30 19:48:04.516714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516660    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce394d1-3507-4d6a-80ef-cfa36122fc07" containerName="revoke-keys"
Oct 30 19:48:04.516714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516683    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1736694d-91fc-4895-8339-7754f5996c3a" containerName="revoke-keys"
Oct 30 19:48:04.516714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516691    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1736694d-91fc-4895-8339-7754f5996c3a" containerName="revoke-keys"
Oct 30 19:48:04.516826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516734    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d9e19118-1d38-4faa-9543-4e17c51cad2c" containerName="revoke-keys"
Oct 30 19:48:04.516826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516743    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e19118-1d38-4faa-9543-4e17c51cad2c" containerName="revoke-keys"
Oct 30 19:48:04.516892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516865    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce394d1-3507-4d6a-80ef-cfa36122fc07" containerName="revoke-keys"
Oct 30 19:48:04.516892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516883    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1736694d-91fc-4895-8339-7754f5996c3a" containerName="revoke-keys"
Oct 30 19:48:04.516961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.516898    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e19118-1d38-4faa-9543-4e17c51cad2c" containerName="revoke-keys"
Oct 30 19:48:04.520788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.520771    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.574054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.574018    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"]
Oct 30 19:48:04.615792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.615760    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:04.615945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.615816    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:48:04.634920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.634884    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.635067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.634963    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.635106 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.635070    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg5n\" (UniqueName: \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.738576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.737366    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.738576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.737518    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.738576 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.737635    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg5n\" (UniqueName: \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.738861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.738597    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.741593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.741501    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.797139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.797069    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg5n\" (UniqueName: \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\") pod \"maas-api-revoke-sub-2552377d-xlx6g\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:04.834520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:04.834493    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:05.051176 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.051080    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"]
Oct 30 19:48:05.055142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.055119    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39879b3d-x5b5f"]
Oct 30 19:48:05.206923 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:05.206882    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4363df01_af0f_417e_b8b0_21f04631476a.slice/crio-dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2 WatchSource:0}: Error finding container dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2: Status 404 returned error can't find the container with id dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2
Oct 30 19:48:05.207162 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.207139    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"]
Oct 30 19:48:05.752647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.752608    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g" event={"ID":"4363df01-af0f-417e-b8b0-21f04631476a","Type":"ContainerStarted","Data":"a15285ae4c72e5827bb2063868e74aee9c95d7196c8d4cbb03112c9c4a7e6fb5"}
Oct 30 19:48:05.753011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.752667    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g" event={"ID":"4363df01-af0f-417e-b8b0-21f04631476a","Type":"ContainerStarted","Data":"dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2"}
Oct 30 19:48:05.759771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.759744    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"]
Oct 30 19:48:05.764123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.764105    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.822854 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.822827    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"]
Oct 30 19:48:05.852303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.852266    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.852496 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.852475    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.852557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.852545    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgch\" (UniqueName: \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.867220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.867167    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g" podStartSLOduration=1.8671548040000001 podStartE2EDuration="1.867154804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:05.862894181 +0000 UTC m=+1395.367297929" watchObservedRunningTime="2026-10-30 19:48:05.867154804 +0000 UTC m=+1395.371558565"
Oct 30 19:48:05.953594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.953546    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.953766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.953625    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgch\" (UniqueName: \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.953825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.953777    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.954299 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.954278    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:05.956219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:05.956199    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:06.012365 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.012288    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgch\" (UniqueName: \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\") pod \"maas-api-revoke-sub-33cf573d-c5r62\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:06.050112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.050081    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"]
Oct 30 19:48:06.054132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.054106    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a1b59fa-x5s7b"]
Oct 30 19:48:06.078258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.078235    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:06.454177 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.454147    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"]
Oct 30 19:48:06.454319 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:06.454149    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec54adc_a1a9_401b_af4b_05e7f6d86fac.slice/crio-74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17 WatchSource:0}: Error finding container 74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17: Status 404 returned error can't find the container with id 74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17
Oct 30 19:48:06.633262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.633221    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ae6626-147a-4872-8088-0a7ae45ee83d" path="/var/lib/kubelet/pods/79ae6626-147a-4872-8088-0a7ae45ee83d/volumes"
Oct 30 19:48:06.633832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.633810    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b88d4f-b55f-4f21-9690-95b4b7b0555b" path="/var/lib/kubelet/pods/c9b88d4f-b55f-4f21-9690-95b4b7b0555b/volumes"
Oct 30 19:48:06.761446 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.761413    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62" event={"ID":"fec54adc-a1a9-401b-af4b-05e7f6d86fac","Type":"ContainerStarted","Data":"74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17"}
Oct 30 19:48:06.763387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.763358    5085 generic.go:356] "Generic (PLEG): container finished" podID="4363df01-af0f-417e-b8b0-21f04631476a" containerID="a15285ae4c72e5827bb2063868e74aee9c95d7196c8d4cbb03112c9c4a7e6fb5" exitCode=0
Oct 30 19:48:06.763517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:06.763394    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g" event={"ID":"4363df01-af0f-417e-b8b0-21f04631476a","Type":"ContainerDied","Data":"a15285ae4c72e5827bb2063868e74aee9c95d7196c8d4cbb03112c9c4a7e6fb5"}
Oct 30 19:48:07.780259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:07.780219    5085 generic.go:356] "Generic (PLEG): container finished" podID="fec54adc-a1a9-401b-af4b-05e7f6d86fac" containerID="e0faeef05e9b1afcc2e06dbdbd352aa9e88375501c0e38f8bf3d6e415b3834d4" exitCode=0
Oct 30 19:48:07.780760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:07.780593    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62" event={"ID":"fec54adc-a1a9-401b-af4b-05e7f6d86fac","Type":"ContainerDied","Data":"e0faeef05e9b1afcc2e06dbdbd352aa9e88375501c0e38f8bf3d6e415b3834d4"}
Oct 30 19:48:07.939931 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:07.939906    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:08.082388 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.082309    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\") pod \"4363df01-af0f-417e-b8b0-21f04631476a\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") "
Oct 30 19:48:08.082524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.082504    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\") pod \"4363df01-af0f-417e-b8b0-21f04631476a\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") "
Oct 30 19:48:08.082604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.082531    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\" (UniqueName: \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\") pod \"4363df01-af0f-417e-b8b0-21f04631476a\" (UID: \"4363df01-af0f-417e-b8b0-21f04631476a\") "
Oct 30 19:48:08.082978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.082942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca" pod "4363df01-af0f-417e-b8b0-21f04631476a" (UID: "4363df01-af0f-417e-b8b0-21f04631476a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:08.084488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.084459    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca" pod "4363df01-af0f-417e-b8b0-21f04631476a" (UID: "4363df01-af0f-417e-b8b0-21f04631476a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:08.085104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.085083    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n" pod "4363df01-af0f-417e-b8b0-21f04631476a" (UID: "4363df01-af0f-417e-b8b0-21f04631476a"). InnerVolumeSpecName "kube-api-access-7tg5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:08.184300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.184262    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:08.184300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.184296    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tg5n\" (UniqueName: \"kubernetes.io/projected/4363df01-af0f-417e-b8b0-21f04631476a-kube-api-access-7tg5n\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:08.184300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.184307    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4363df01-af0f-417e-b8b0-21f04631476a-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:08.738557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.738525    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 19:48:08.739186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.739170    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4363df01-af0f-417e-b8b0-21f04631476a" containerName="revoke-keys"
Oct 30 19:48:08.739233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.739187    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4363df01-af0f-417e-b8b0-21f04631476a" containerName="revoke-keys"
Oct 30 19:48:08.739315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.739304    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4363df01-af0f-417e-b8b0-21f04631476a" containerName="revoke-keys"
Oct 30 19:48:08.742941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.742923    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.789950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.789916    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g" event={"ID":"4363df01-af0f-417e-b8b0-21f04631476a","Type":"ContainerDied","Data":"dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2"}
Oct 30 19:48:08.790399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.789971    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc15561f40743e1398d34baa972d2ae2a445edc9151a724013ff6d8519709bc2"
Oct 30 19:48:08.790399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.789991    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"
Oct 30 19:48:08.840998 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.840969    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b0a183\"/\"distinct-2-w2-b0a183-kserve-self-signed-certs\""
Oct 30 19:48:08.845398 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.845370    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893412    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893629    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893719    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gkqm\" (UniqueName: \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893899    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893943    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.894269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.893982    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.993827 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.993803    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:08.995103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995077    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gkqm\" (UniqueName: \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995197    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995223    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995253    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995303    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995340    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995644    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995774    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.995807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.995798    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.997534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.997515    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:08.998109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:08.998089    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:09.053572 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.053534    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gkqm\" (UniqueName: \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\") pod \"distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") " pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:09.055241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.055218    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:09.096803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.096766    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\" (UniqueName: \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\") pod \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") "
Oct 30 19:48:09.097043 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.097025    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\") pod \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") "
Oct 30 19:48:09.097154 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.097096    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\") pod \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\" (UID: \"fec54adc-a1a9-401b-af4b-05e7f6d86fac\") "
Oct 30 19:48:09.097555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.097528    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca" pod "fec54adc-a1a9-401b-af4b-05e7f6d86fac" (UID: "fec54adc-a1a9-401b-af4b-05e7f6d86fac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:09.097699 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.097652    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:09.099753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.099727    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch" pod "fec54adc-a1a9-401b-af4b-05e7f6d86fac" (UID: "fec54adc-a1a9-401b-af4b-05e7f6d86fac"). InnerVolumeSpecName "kube-api-access-mfgch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:09.100120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.100099    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca" pod "fec54adc-a1a9-401b-af4b-05e7f6d86fac" (UID: "fec54adc-a1a9-401b-af4b-05e7f6d86fac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:09.198893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.198861    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec54adc-a1a9-401b-af4b-05e7f6d86fac-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:09.198893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.198889    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfgch\" (UniqueName: \"kubernetes.io/projected/fec54adc-a1a9-401b-af4b-05e7f6d86fac-kube-api-access-mfgch\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:09.435202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.435171    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 19:48:09.438801 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:09.438102    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dfd1e13_ef55_4072_9a50_468802aa4fc1.slice/crio-600861b15327c0e24eaabc10909940def6fdcf39f164522c6e72224b6e87520a WatchSource:0}: Error finding container 600861b15327c0e24eaabc10909940def6fdcf39f164522c6e72224b6e87520a: Status 404 returned error can't find the container with id 600861b15327c0e24eaabc10909940def6fdcf39f164522c6e72224b6e87520a
Oct 30 19:48:09.798498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.798420    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"
Oct 30 19:48:09.798498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.798446    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62" event={"ID":"fec54adc-a1a9-401b-af4b-05e7f6d86fac","Type":"ContainerDied","Data":"74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17"}
Oct 30 19:48:09.798498 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.798493    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74faaad60d99885028bc9a5541c751bb16f8208bf2f0e9b31ceb3d9210148c17"
Oct 30 19:48:09.800442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.800417    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" event={"ID":"5dfd1e13-ef55-4072-9a50-468802aa4fc1","Type":"ContainerStarted","Data":"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"}
Oct 30 19:48:09.800548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.800449    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" event={"ID":"5dfd1e13-ef55-4072-9a50-468802aa4fc1","Type":"ContainerStarted","Data":"600861b15327c0e24eaabc10909940def6fdcf39f164522c6e72224b6e87520a"}
Oct 30 19:48:09.918155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:09.918104    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podStartSLOduration=1.91808918 podStartE2EDuration="1.91808918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:09.916093883 +0000 UTC m=+1399.420497630" watchObservedRunningTime="2026-10-30 19:48:09.91808918 +0000 UTC m=+1399.422492926"
Oct 30 19:48:10.055729 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:10.055639    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:13.202886 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.202851    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"]
Oct 30 19:48:13.203613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.203594    5085 cpu_manager.go:414] "containerMap: removing container" podUID="fec54adc-a1a9-401b-af4b-05e7f6d86fac" containerName="revoke-keys"
Oct 30 19:48:13.203675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.203614    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec54adc-a1a9-401b-af4b-05e7f6d86fac" containerName="revoke-keys"
Oct 30 19:48:13.203728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.203719    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec54adc-a1a9-401b-af4b-05e7f6d86fac" containerName="revoke-keys"
Oct 30 19:48:13.208844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.208824    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.286794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.286759    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"]
Oct 30 19:48:13.291002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.290971    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5tv6\" (UniqueName: \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.291169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.291045    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.291235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.291180    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.327782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.327747    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:13.327949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.327888    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:48:13.392189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.392155    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5tv6\" (UniqueName: \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.392390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.392227    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.392390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.392368    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.393071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.393044    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.394819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.394797    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.450547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.450517    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5tv6\" (UniqueName: \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\") pod \"maas-api-revoke-sub-8f56c068-xm6hb\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.523013 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.522933    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:13.900604 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:13.900546    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode154bcfe_a1d0_4323_8b5a_a5c4e7e3a705.slice/crio-d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2 WatchSource:0}: Error finding container d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2: Status 404 returned error can't find the container with id d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2
Oct 30 19:48:13.901033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:13.901012    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"]
Oct 30 19:48:14.903480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:14.903449    5085 generic.go:356] "Generic (PLEG): container finished" podID="e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" containerID="56d4f9252567b57963ce1b0d84dac3ad2e387e3ec01eadb1e0eee7aee8b04e78" exitCode=0
Oct 30 19:48:14.903898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:14.903513    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb" event={"ID":"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705","Type":"ContainerDied","Data":"56d4f9252567b57963ce1b0d84dac3ad2e387e3ec01eadb1e0eee7aee8b04e78"}
Oct 30 19:48:14.903898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:14.903595    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb" event={"ID":"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705","Type":"ContainerStarted","Data":"d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2"}
Oct 30 19:48:16.068644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.068616    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:16.121646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.121615    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\") pod \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") "
Oct 30 19:48:16.121814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.121673    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\") pod \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") "
Oct 30 19:48:16.121814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.121694    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\" (UniqueName: \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\") pod \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\" (UID: \"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705\") "
Oct 30 19:48:16.122020 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.121997    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca" pod "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" (UID: "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:16.122243 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.122227    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:16.123779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.123759    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca" pod "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" (UID: "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:16.123847 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.123764    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6" pod "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" (UID: "e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705"). InnerVolumeSpecName "kube-api-access-w5tv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:16.223668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.223634    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:16.223668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.223665    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5tv6\" (UniqueName: \"kubernetes.io/projected/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705-kube-api-access-w5tv6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:16.921195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.921165    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"
Oct 30 19:48:16.921195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.921179    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb" event={"ID":"e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705","Type":"ContainerDied","Data":"d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2"}
Oct 30 19:48:16.921424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:16.921229    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35f2f705998d2d4693b0f2d5b1bcf253011c57ffc3c3228dffd2791e1b2cef2"
Oct 30 19:48:19.085587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:19.064941    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:19.386642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:19.385774    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:19.451590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:19.449648    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.067696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.067646    5085 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.136587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.135668    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.137099 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.136991    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.393639 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.393255    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:20.416589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.411269    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:48:20.416589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.411864    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:20.957697 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:20.957650    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded"
Oct 30 19:48:21.181031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.155021    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:21.181031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.156822    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:21.181031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.157895    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:21.188810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.188760    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:48:21.394583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.394479    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"]
Oct 30 19:48:21.394938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.394838    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:21.448589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.445042    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:48:21.449356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.449312    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:48:21.449634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.449610    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:48:21.512277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:21.498401    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25916885-9pdd2"]
Oct 30 19:48:22.213648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.168044    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:22.302882 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.302836    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:22.398594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.380236    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:48:22.412818 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.412770    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:22.448005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.447951    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:22.617647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.617431    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:48:22.617647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:22.457909    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:23.199611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.186644    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:23.223881 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.220869    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c317712-7d83-4e7d-9ad1-78a9af22da5e" path="/var/lib/kubelet/pods/7c317712-7d83-4e7d-9ad1-78a9af22da5e/volumes"
Oct 30 19:48:23.401590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.381644    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:48:23.436950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.436910    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:23.561206 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:48:23.494825    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:48:23.561206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.521803    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:23.630889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:23.630842    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.075727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.075678    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.189714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.189665    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.396198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.396064    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.471624 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.462179    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:48:24.542726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.542683    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.736603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.733740    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:24.813577 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.813526    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:24.813850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.813829    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:48:24.818909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.818878    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:48:24.839036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:24.839008    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:48:26.057710 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.057676    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"]
Oct 30 19:48:26.104347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.104314    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2babaa97-n2hr2"]
Oct 30 19:48:26.180319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.180280    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"]
Oct 30 19:48:26.180912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.180895    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" containerName="revoke-keys"
Oct 30 19:48:26.181010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.180913    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" containerName="revoke-keys"
Oct 30 19:48:26.181082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.181048    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" containerName="revoke-keys"
Oct 30 19:48:26.187471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.187439    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.237681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.237654    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:26.237862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.237845    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:48:26.242261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.242235    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"]
Oct 30 19:48:26.340423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.340347    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2vb\" (UniqueName: \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.340593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.340467    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.340593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.340508    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.441681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.441643    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2vb\" (UniqueName: \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.441869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.441730    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.441869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.441765    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.442349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.442327    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.444093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.444071    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.500234 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.500207    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2vb\" (UniqueName: \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\") pod \"maas-api-revoke-sub-ccd7852e-n5pzb\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.504057 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.504031    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:26.633129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.633035    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6670f79d-9ad6-451e-8e57-eb5897307d6d" path="/var/lib/kubelet/pods/6670f79d-9ad6-451e-8e57-eb5897307d6d/volumes"
Oct 30 19:48:26.880329 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:26.880304    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebbd892_93d5_42b4_ab9c_cc944e114841.slice/crio-52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d WatchSource:0}: Error finding container 52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d: Status 404 returned error can't find the container with id 52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d
Oct 30 19:48:26.881123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:26.880995    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"]
Oct 30 19:48:27.268622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:27.268558    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb" event={"ID":"1ebbd892-93d5-42b4-ab9c-cc944e114841","Type":"ContainerStarted","Data":"3a66c08afbd5ce4d2b2b563bc6d24befaf2ed7b05fb68c531fc24eb7c74b47e9"}
Oct 30 19:48:27.269015 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:27.268650    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb" event={"ID":"1ebbd892-93d5-42b4-ab9c-cc944e114841","Type":"ContainerStarted","Data":"52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d"}
Oct 30 19:48:27.379734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:27.379679    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb" podStartSLOduration=1.379666415 podStartE2EDuration="1.379666415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:27.378778949 +0000 UTC m=+1416.883182697" watchObservedRunningTime="2026-10-30 19:48:27.379666415 +0000 UTC m=+1416.884070162"
Oct 30 19:48:28.277954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:28.277917    5085 generic.go:356] "Generic (PLEG): container finished" podID="1ebbd892-93d5-42b4-ab9c-cc944e114841" containerID="3a66c08afbd5ce4d2b2b563bc6d24befaf2ed7b05fb68c531fc24eb7c74b47e9" exitCode=0
Oct 30 19:48:28.278349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:28.277951    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb" event={"ID":"1ebbd892-93d5-42b4-ab9c-cc944e114841","Type":"ContainerDied","Data":"3a66c08afbd5ce4d2b2b563bc6d24befaf2ed7b05fb68c531fc24eb7c74b47e9"}
Oct 30 19:48:29.067528 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.067498    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:29.079068 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.079038    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:48:29.481466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.481443    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:29.589382 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.589345    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\") pod \"1ebbd892-93d5-42b4-ab9c-cc944e114841\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") "
Oct 30 19:48:29.589648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.589468    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\" (UniqueName: \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\") pod \"1ebbd892-93d5-42b4-ab9c-cc944e114841\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") "
Oct 30 19:48:29.589648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.589605    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\") pod \"1ebbd892-93d5-42b4-ab9c-cc944e114841\" (UID: \"1ebbd892-93d5-42b4-ab9c-cc944e114841\") "
Oct 30 19:48:29.590252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.590222    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca" pod "1ebbd892-93d5-42b4-ab9c-cc944e114841" (UID: "1ebbd892-93d5-42b4-ab9c-cc944e114841"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:29.592875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.592825    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca" pod "1ebbd892-93d5-42b4-ab9c-cc944e114841" (UID: "1ebbd892-93d5-42b4-ab9c-cc944e114841"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:29.592875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.592858    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb" pod "1ebbd892-93d5-42b4-ab9c-cc944e114841" (UID: "1ebbd892-93d5-42b4-ab9c-cc944e114841"). InnerVolumeSpecName "kube-api-access-cj2vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:29.690780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.690744    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.690780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.690776    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ebbd892-93d5-42b4-ab9c-cc944e114841-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.690780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.690788    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2vb\" (UniqueName: \"kubernetes.io/projected/1ebbd892-93d5-42b4-ab9c-cc944e114841-kube-api-access-cj2vb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:29.957701 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.957623    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"]
Oct 30 19:48:29.958262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.958246    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1ebbd892-93d5-42b4-ab9c-cc944e114841" containerName="revoke-keys"
Oct 30 19:48:29.958262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.958262    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebbd892-93d5-42b4-ab9c-cc944e114841" containerName="revoke-keys"
Oct 30 19:48:29.958386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.958370    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebbd892-93d5-42b4-ab9c-cc944e114841" containerName="revoke-keys"
Oct 30 19:48:29.962203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:29.962182    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.017343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.017313    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"]
Oct 30 19:48:30.095674 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.095634    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.095824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.095680    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46q4x\" (UniqueName: \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.095881 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.095824    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.197594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.197546    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.197772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.197612    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46q4x\" (UniqueName: \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.197772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.197716    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.198370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.198342    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.199831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.199811    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.255800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.255740    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46q4x\" (UniqueName: \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\") pod \"maas-api-revoke-sub-5d637a5e-r22r5\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.285707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.285680    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:30.301936 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.301916    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"
Oct 30 19:48:30.302045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.301960    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb" event={"ID":"1ebbd892-93d5-42b4-ab9c-cc944e114841","Type":"ContainerDied","Data":"52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d"}
Oct 30 19:48:30.302045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.302020    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52354d2946947003ff3c1864b2a38514d82eb9cfc1520b591992883b9296fc9d"
Oct 30 19:48:30.659781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:30.659753    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"]
Oct 30 19:48:30.660751 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:30.660725    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c303019_523f_4bd8_b451_838e39718005.slice/crio-479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac WatchSource:0}: Error finding container 479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac: Status 404 returned error can't find the container with id 479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac
Oct 30 19:48:31.312300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:31.312214    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5" event={"ID":"2c303019-523f-4bd8-b451-838e39718005","Type":"ContainerStarted","Data":"0603f569e43af41fd1eb0a5562bf84e3f18c6d0b98a50931cd43f89d20b5c112"}
Oct 30 19:48:31.312300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:31.312276    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5" event={"ID":"2c303019-523f-4bd8-b451-838e39718005","Type":"ContainerStarted","Data":"479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac"}
Oct 30 19:48:32.321534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:32.321496    5085 generic.go:356] "Generic (PLEG): container finished" podID="2c303019-523f-4bd8-b451-838e39718005" containerID="0603f569e43af41fd1eb0a5562bf84e3f18c6d0b98a50931cd43f89d20b5c112" exitCode=0
Oct 30 19:48:32.321949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:32.321591    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5" event={"ID":"2c303019-523f-4bd8-b451-838e39718005","Type":"ContainerDied","Data":"0603f569e43af41fd1eb0a5562bf84e3f18c6d0b98a50931cd43f89d20b5c112"}
Oct 30 19:48:33.503788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.503762    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:33.536746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.536719    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\" (UniqueName: \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\") pod \"2c303019-523f-4bd8-b451-838e39718005\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") "
Oct 30 19:48:33.536868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.536791    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\") pod \"2c303019-523f-4bd8-b451-838e39718005\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") "
Oct 30 19:48:33.536868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.536821    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\") pod \"2c303019-523f-4bd8-b451-838e39718005\" (UID: \"2c303019-523f-4bd8-b451-838e39718005\") "
Oct 30 19:48:33.537220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.537197    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca" pod "2c303019-523f-4bd8-b451-838e39718005" (UID: "2c303019-523f-4bd8-b451-838e39718005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:33.537363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.537350    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c303019-523f-4bd8-b451-838e39718005-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:33.538804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.538778    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca" pod "2c303019-523f-4bd8-b451-838e39718005" (UID: "2c303019-523f-4bd8-b451-838e39718005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:33.538918 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.538832    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x" pod "2c303019-523f-4bd8-b451-838e39718005" (UID: "2c303019-523f-4bd8-b451-838e39718005"). InnerVolumeSpecName "kube-api-access-46q4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:33.638529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.638258    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46q4x\" (UniqueName: \"kubernetes.io/projected/2c303019-523f-4bd8-b451-838e39718005-kube-api-access-46q4x\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:33.638529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:33.638291    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c303019-523f-4bd8-b451-838e39718005-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:34.338679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:34.338649    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"
Oct 30 19:48:34.338875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:34.338652    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5" event={"ID":"2c303019-523f-4bd8-b451-838e39718005","Type":"ContainerDied","Data":"479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac"}
Oct 30 19:48:34.338875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:34.338783    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479655a833f2dec60d7a5e2bcfe1e3b9544896d1ca596bf2db0db44577eb04ac"
Oct 30 19:48:36.059535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.059484    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"]
Oct 30 19:48:36.098947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.098888    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80269d8c-6wx5d"]
Oct 30 19:48:36.149416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.149381    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:48:36.150186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.150151    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2c303019-523f-4bd8-b451-838e39718005" containerName="revoke-keys"
Oct 30 19:48:36.150186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.150183    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c303019-523f-4bd8-b451-838e39718005" containerName="revoke-keys"
Oct 30 19:48:36.150395 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.150347    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c303019-523f-4bd8-b451-838e39718005" containerName="revoke-keys"
Oct 30 19:48:36.153655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.153634    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.169677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169648    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.169805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169709    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.169846 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169799    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.169846 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169827    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.169954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169910    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7rl\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.170008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.169996    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.170049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.170038    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.170090 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.170059    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.170148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.170134    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.237423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.237390    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:48:36.271039 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271007    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7rl\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271071    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271112    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271129    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271282    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271391    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271458    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271507    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271531    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271604    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271668    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.271954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271931    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.272042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.271999    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.272555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.272527    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.273877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.273850    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.274216 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.274192    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.329522 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.329451    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.329877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.329856    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7rl\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.467539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.467508    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:36.632175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.632095    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b560f1-9897-42a0-aa8e-71a1809835cf" path="/var/lib/kubelet/pods/94b560f1-9897-42a0-aa8e-71a1809835cf/volumes"
Oct 30 19:48:36.896849 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:36.896780    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ad8d66_5114_4efc_a464_0132276abba3.slice/crio-1c6fdf9bcb6caba95e7c285f0e75ecb3eba0239dc64767e6e559d1f9bbf3c164 WatchSource:0}: Error finding container 1c6fdf9bcb6caba95e7c285f0e75ecb3eba0239dc64767e6e559d1f9bbf3c164: Status 404 returned error can't find the container with id 1c6fdf9bcb6caba95e7c285f0e75ecb3eba0239dc64767e6e559d1f9bbf3c164
Oct 30 19:48:36.896982 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.896896    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:48:36.899278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.899245    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:36.899355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.899308    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:36.899355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.899338    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:36.957259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.957236    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"]
Oct 30 19:48:36.961258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.961236    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:36.982997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.982963    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:36.983194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.983170    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:36.983297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:36.983274    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58h2h\" (UniqueName: \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.017610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.015821    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:37.017610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.015942    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:48:37.019025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.018998    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"]
Oct 30 19:48:37.085014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.084981    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.085481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.085085    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.085481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.085128    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58h2h\" (UniqueName: \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.085795 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.085773    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.087282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.087262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.144055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.143972    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58h2h\" (UniqueName: \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\") pod \"maas-api-revoke-sub-f3bc3a4a-7bvl6\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.275329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.275298    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:37.364943 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.364909    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" event={"ID":"a3ad8d66-5114-4efc-a464-0132276abba3","Type":"ContainerStarted","Data":"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"}
Oct 30 19:48:37.365112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.364968    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" event={"ID":"a3ad8d66-5114-4efc-a464-0132276abba3","Type":"ContainerStarted","Data":"1c6fdf9bcb6caba95e7c285f0e75ecb3eba0239dc64767e6e559d1f9bbf3c164"}
Oct 30 19:48:37.468408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.468375    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:37.470109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.470090    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:37.470171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.470136    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:37.484072 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.484024    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podStartSLOduration=1.484010839 podStartE2EDuration="1.484010839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:37.479558045 +0000 UTC m=+1426.983961793" watchObservedRunningTime="2026-10-30 19:48:37.484010839 +0000 UTC m=+1426.988414585"
Oct 30 19:48:37.660274 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:37.660231    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"]
Oct 30 19:48:37.661900 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:37.661867    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f65066_7f6e_4c76_bf09_53b4f6d61dcc.slice/crio-bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73 WatchSource:0}: Error finding container bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73: Status 404 returned error can't find the container with id bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73
Oct 30 19:48:38.392268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:38.392173    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6" event={"ID":"85f65066-7f6e-4c76-bf09-53b4f6d61dcc","Type":"ContainerStarted","Data":"a20fa2d02969951c8f3e92568ebf22c0ae1983cd0288ca6e9e498e627ada9465"}
Oct 30 19:48:38.392827 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:38.392272    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6" event={"ID":"85f65066-7f6e-4c76-bf09-53b4f6d61dcc","Type":"ContainerStarted","Data":"bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73"}
Oct 30 19:48:38.479586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:38.473988    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:38.479586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:38.474064    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:38.508489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:38.508432    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6" podStartSLOduration=2.508414024 podStartE2EDuration="2.508414024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:38.503367468 +0000 UTC m=+1428.007771215" watchObservedRunningTime="2026-10-30 19:48:38.508414024 +0000 UTC m=+1428.012817773"
Oct 30 19:48:39.402416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:39.402377    5085 generic.go:356] "Generic (PLEG): container finished" podID="85f65066-7f6e-4c76-bf09-53b4f6d61dcc" containerID="a20fa2d02969951c8f3e92568ebf22c0ae1983cd0288ca6e9e498e627ada9465" exitCode=0
Oct 30 19:48:39.402895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:39.402476    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6" event={"ID":"85f65066-7f6e-4c76-bf09-53b4f6d61dcc","Type":"ContainerDied","Data":"a20fa2d02969951c8f3e92568ebf22c0ae1983cd0288ca6e9e498e627ada9465"}
Oct 30 19:48:39.483054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:39.483014    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:39.483286 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:39.483084    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:39.787226 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:39.787185    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:48:40.491588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:40.484901    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:40.491588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:40.484957    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:40.917653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:40.916342    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"]
Oct 30 19:48:40.950585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:40.938628    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.015598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:40.999748    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"]
Oct 30 19:48:41.140585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.140439    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.140915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.140669    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnff\" (UniqueName: \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.140915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.140725    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.281380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.277507    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.281380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.277742    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnff\" (UniqueName: \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.281380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.277801    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.327003 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.324147    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.409590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.407647    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:41.430593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.430102    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.452186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.448812    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:48:41.470367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.470331    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnff\" (UniqueName: \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\") pod \"maas-api-revoke-sub-d3d35bd2-k4xpj\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.472881 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.472658    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:41.473297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.473271    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:41.557833 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.557736    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:41.692128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.692097    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:41.692594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:41.692549    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.150734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.150700    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:42.334587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.332876    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\" (UniqueName: \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\") pod \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") "
Oct 30 19:48:42.334587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.333040    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\") pod \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") "
Oct 30 19:48:42.334587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.333178    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\") pod \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\" (UID: \"85f65066-7f6e-4c76-bf09-53b4f6d61dcc\") "
Oct 30 19:48:42.351999 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.347023    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.404613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.380502    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.406279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.394094    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca" pod "85f65066-7f6e-4c76-bf09-53b4f6d61dcc" (UID: "85f65066-7f6e-4c76-bf09-53b4f6d61dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:42.410584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.410067    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.410871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.410849    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:42.485587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.485004    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca" pod "85f65066-7f6e-4c76-bf09-53b4f6d61dcc" (UID: "85f65066-7f6e-4c76-bf09-53b4f6d61dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:42.509587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.503413    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:42.509587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.503474    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:42.509587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.503656    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h" pod "85f65066-7f6e-4c76-bf09-53b4f6d61dcc" (UID: "85f65066-7f6e-4c76-bf09-53b4f6d61dcc"). InnerVolumeSpecName "kube-api-access-58h2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:42.555589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.545710    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.555589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.546099    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:48:42.555589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.552556    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.555589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.552607    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58h2h\" (UniqueName: \"kubernetes.io/projected/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-kube-api-access-58h2h\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.555589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.552626    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85f65066-7f6e-4c76-bf09-53b4f6d61dcc-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:42.655960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.655922    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:42.689699 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.689650    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.775642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.765471    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:42.902851 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.902821    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:48:42.953523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.944245    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:48:42.990146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.971629    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6" event={"ID":"85f65066-7f6e-4c76-bf09-53b4f6d61dcc","Type":"ContainerDied","Data":"bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73"}
Oct 30 19:48:42.990146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.971688    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb4a8fcd186c4ae5b6c712be9a35f977a068781e458755f1671777b9ef528f73"
Oct 30 19:48:42.990146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:42.963623    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"
Oct 30 19:48:43.111324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.111240    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:48:43.120229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.120206    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:43.319510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.319479    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"]
Oct 30 19:48:43.320254 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:43.320228    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58ecf83_180a_4ec7_9e94_65d20827e972.slice/crio-3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650 WatchSource:0}: Error finding container 3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650: Status 404 returned error can't find the container with id 3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650
Oct 30 19:48:43.487797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.487758    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:43.487977 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.487826    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:43.990654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.990611    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj" event={"ID":"e58ecf83-180a-4ec7-9e94-65d20827e972","Type":"ContainerStarted","Data":"ed368d7fe8cce8380531664e1ced3fcdd508736cf3506dfa53791cd92380ad20"}
Oct 30 19:48:43.991119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:43.990678    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj" event={"ID":"e58ecf83-180a-4ec7-9e94-65d20827e972","Type":"ContainerStarted","Data":"3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650"}
Oct 30 19:48:44.103257 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:44.103153    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj" podStartSLOduration=4.103138256 podStartE2EDuration="4.103138256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:44.100279627 +0000 UTC m=+1433.604683371" watchObservedRunningTime="2026-10-30 19:48:44.103138256 +0000 UTC m=+1433.607542002"
Oct 30 19:48:44.468455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:44.468419    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body=
Oct 30 19:48:44.468626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:44.468483    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused"
Oct 30 19:48:45.000858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:45.000820    5085 generic.go:356] "Generic (PLEG): container finished" podID="e58ecf83-180a-4ec7-9e94-65d20827e972" containerID="ed368d7fe8cce8380531664e1ced3fcdd508736cf3506dfa53791cd92380ad20" exitCode=0
Oct 30 19:48:45.001248 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:45.000923    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj" event={"ID":"e58ecf83-180a-4ec7-9e94-65d20827e972","Type":"ContainerDied","Data":"ed368d7fe8cce8380531664e1ced3fcdd508736cf3506dfa53791cd92380ad20"}
Oct 30 19:48:46.238460 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.238433    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:46.379904 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.379807    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\") pod \"e58ecf83-180a-4ec7-9e94-65d20827e972\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") "
Oct 30 19:48:46.380085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.379924    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\" (UniqueName: \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\") pod \"e58ecf83-180a-4ec7-9e94-65d20827e972\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") "
Oct 30 19:48:46.380085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.380032    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\") pod \"e58ecf83-180a-4ec7-9e94-65d20827e972\" (UID: \"e58ecf83-180a-4ec7-9e94-65d20827e972\") "
Oct 30 19:48:46.380208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.380156    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca" pod "e58ecf83-180a-4ec7-9e94-65d20827e972" (UID: "e58ecf83-180a-4ec7-9e94-65d20827e972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:46.381277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.381252    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:46.383095 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.383058    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff" pod "e58ecf83-180a-4ec7-9e94-65d20827e972" (UID: "e58ecf83-180a-4ec7-9e94-65d20827e972"). InnerVolumeSpecName "kube-api-access-ddnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:46.383654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.383629    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca" pod "e58ecf83-180a-4ec7-9e94-65d20827e972" (UID: "e58ecf83-180a-4ec7-9e94-65d20827e972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:48:46.483256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.483218    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58ecf83-180a-4ec7-9e94-65d20827e972-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:46.483256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.483256    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnff\" (UniqueName: \"kubernetes.io/projected/e58ecf83-180a-4ec7-9e94-65d20827e972-kube-api-access-ddnff\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:48:46.508769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.508724    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:48:46.508979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.508796    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:46.775135 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.775104    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:46.775815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.775787    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:46.806010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:46.805897    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:48:47.022819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:47.022787    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"
Oct 30 19:48:47.023027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:47.022815    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj" event={"ID":"e58ecf83-180a-4ec7-9e94-65d20827e972","Type":"ContainerDied","Data":"3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650"}
Oct 30 19:48:47.023027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:47.022867    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a67b4e109b00a7e42cf0ed00bcc9833dbe7c616adfb00cc99ae2a833159e650"
Oct 30 19:48:51.168884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.168850    5085 scope.go:122] "RemoveContainer" containerID="59dfeaa51a2d6ea71d34015d6aeba2f96038931e602e64bb6abcf27ebd83e92a"
Oct 30 19:48:51.180353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.180329    5085 scope.go:122] "RemoveContainer" containerID="2a3892ea923fa0ad2d3205a59f962a646f4803ac0716016338dc28b0343467d5"
Oct 30 19:48:51.281974 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.281939    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:48:51.282631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282612    5085 cpu_manager.go:414] "containerMap: removing container" podUID="85f65066-7f6e-4c76-bf09-53b4f6d61dcc" containerName="revoke-keys"
Oct 30 19:48:51.282732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282631    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f65066-7f6e-4c76-bf09-53b4f6d61dcc" containerName="revoke-keys"
Oct 30 19:48:51.282732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282675    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e58ecf83-180a-4ec7-9e94-65d20827e972" containerName="revoke-keys"
Oct 30 19:48:51.282732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282683    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58ecf83-180a-4ec7-9e94-65d20827e972" containerName="revoke-keys"
Oct 30 19:48:51.282887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282841    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58ecf83-180a-4ec7-9e94-65d20827e972" containerName="revoke-keys"
Oct 30 19:48:51.282887 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.282864    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f65066-7f6e-4c76-bf09-53b4f6d61dcc" containerName="revoke-keys"
Oct 30 19:48:51.288486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.288466    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445159 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445108    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445263    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445302    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445326    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445392    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445723 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445599    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445761    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445804    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.445843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.445821    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxp2\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.492798 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.492770    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:48:51.546678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546647    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.546861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546725    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.546861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546748    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.546861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546763    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.546861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546807    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.546861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546854    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546926    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546956    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.546978    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxp2\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.547208    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.547240    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547468 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.547441    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.547515    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.547633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.547540    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.549233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.549211    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.549407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.549387    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.614350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.614319    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.615295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.615266    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxp2\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.904919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:51.904824    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:51.970590 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:48:51.969773    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:48:52.379352 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:52.379317    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:48:52.410696 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:48:52.410669    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81db512_33f1_40ea_aab2_0197a773fa4e.slice/crio-d73a4b4257492178db8c33ce2b33177630a96c42b4470f58e66703a5debc6579 WatchSource:0}: Error finding container d73a4b4257492178db8c33ce2b33177630a96c42b4470f58e66703a5debc6579: Status 404 returned error can't find the container with id d73a4b4257492178db8c33ce2b33177630a96c42b4470f58e66703a5debc6579
Oct 30 19:48:52.413210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:52.413178    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:52.413281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:52.413251    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:52.413323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:52.413279    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:48:53.081341 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.081306    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" event={"ID":"c81db512-33f1-40ea-aab2-0197a773fa4e","Type":"ContainerStarted","Data":"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"}
Oct 30 19:48:53.081517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.081361    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" event={"ID":"c81db512-33f1-40ea-aab2-0197a773fa4e","Type":"ContainerStarted","Data":"d73a4b4257492178db8c33ce2b33177630a96c42b4470f58e66703a5debc6579"}
Oct 30 19:48:53.299400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.299347    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podStartSLOduration=2.299333371 podStartE2EDuration="2.299333371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:53.251065808 +0000 UTC m=+1442.755469554" watchObservedRunningTime="2026-10-30 19:48:53.299333371 +0000 UTC m=+1442.803737118"
Oct 30 19:48:53.905918 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.905883    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:48:53.907431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.907401    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:53.907583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:53.907471    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:54.906321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:54.906288    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:54.906746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:54.906353    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:55.906006 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:55.905972    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:55.906186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:55.906037    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:56.906609 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:56.906541    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:56.907004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:56.906641    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:57.907071 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:57.907028    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:57.907679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:57.907643    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:58.104211 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.102646    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"]
Oct 30 19:48:58.106177 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.106148    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6279f074-gmvxf"]
Oct 30 19:48:58.136753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.135647    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:58.223345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.223305    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 19:48:58.421589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.413655    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:58.647649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.646337    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:58.705588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.696013    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca9faf3-066a-4694-9342-edb682351f5f" path="/var/lib/kubelet/pods/eca9faf3-066a-4694-9342-edb682351f5f/volumes"
Oct 30 19:48:58.910481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.909914    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:48:58.910481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:58.909997    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:48:59.086585 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.085660    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Oct 30 19:48:59.157268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.157223    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:59.158622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.158595    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:48:59.159108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.159088    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:59.171634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.170671    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"]
Oct 30 19:48:59.226606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.225992    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:59.267965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.267935    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.437824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.437710    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.438082 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.438053    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwn9\" (UniqueName: \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.438393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.438375    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.504614 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.503974    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:48:59.504614 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.504365    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:48:59.525600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.505623    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:48:59.525600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.267810    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:48:59.525600 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.507158    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"]
Oct 30 19:48:59.628078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.627999    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.628078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.628049    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwn9\" (UniqueName: \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.628222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.628209    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.664855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.664805    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.665288 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.665260    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:48:59.671654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.666075    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:59.681458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.679222    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:48:59.696718 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.696592    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:48:59.910604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:48:59.910536    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwn9\" (UniqueName: \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\") pod \"maas-api-revoke-sub-a7ca52ec-2b65l\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:49:00.124007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.123921    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:49:00.171451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.165707    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:49:00.241582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.228733    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:00.241582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.229118    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:49:00.241582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.233749    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:00.255246 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.255219    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:49:00.261502 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.261473    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:49:00.358912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.343097    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:49:00.371884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.371430    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:49:00.857181 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.857054    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"]
Oct 30 19:49:00.905872 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.905836    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 19:49:00.906002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:00.905901    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 19:49:01.848493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:01.848453    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l" event={"ID":"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf","Type":"ContainerStarted","Data":"dc198700eba6e8013fb825b4856d593f6034e01eadaaf4f5fbb1b48c596c3795"}
Oct 30 19:49:01.848916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:01.848513    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l" event={"ID":"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf","Type":"ContainerStarted","Data":"c78ab348f3053ecda0b7f8e24e4fc9520fcb8e455b16ad0c7f75f01d4e4b08ea"}
Oct 30 19:49:01.963147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:01.963093    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l" podStartSLOduration=2.963078811 podStartE2EDuration="2.963078811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:01.958525973 +0000 UTC m=+1451.462929720" watchObservedRunningTime="2026-10-30 19:49:01.963078811 +0000 UTC m=+1451.467482559"
Oct 30 19:49:02.828447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:02.828407    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:49:02.828950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:02.828890    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:49:02.830414 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:02.830390    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:49:02.858487 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:02.858457    5085 generic.go:356] "Generic (PLEG): container finished" podID="7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" containerID="dc198700eba6e8013fb825b4856d593f6034e01eadaaf4f5fbb1b48c596c3795" exitCode=0
Oct 30 19:49:02.858877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:02.858542    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l" event={"ID":"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf","Type":"ContainerDied","Data":"dc198700eba6e8013fb825b4856d593f6034e01eadaaf4f5fbb1b48c596c3795"}
Oct 30 19:49:04.032822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.032793    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:49:04.117138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.117052    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\" (UniqueName: \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\") pod \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") "
Oct 30 19:49:04.117294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.117182    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\") pod \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") "
Oct 30 19:49:04.117294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.117260    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\") pod \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\" (UID: \"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf\") "
Oct 30 19:49:04.117649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.117620    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca" pod "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" (UID: "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:04.117963 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.117940    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:04.119243 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.119215    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca" pod "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" (UID: "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:04.119323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.119218    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9" pod "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" (UID: "7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf"). InnerVolumeSpecName "kube-api-access-gwwn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:04.219367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.219333    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwwn9\" (UniqueName: \"kubernetes.io/projected/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-kube-api-access-gwwn9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:04.219367 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.219363    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:04.875330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.875297    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"
Oct 30 19:49:04.875504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.875333    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l" event={"ID":"7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf","Type":"ContainerDied","Data":"c78ab348f3053ecda0b7f8e24e4fc9520fcb8e455b16ad0c7f75f01d4e4b08ea"}
Oct 30 19:49:04.875504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:04.875383    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78ab348f3053ecda0b7f8e24e4fc9520fcb8e455b16ad0c7f75f01d4e4b08ea"
Oct 30 19:49:08.915951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.915912    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"]
Oct 30 19:49:08.916540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.916524    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" containerName="revoke-keys"
Oct 30 19:49:08.916641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.916541    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" containerName="revoke-keys"
Oct 30 19:49:08.916710 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.916698    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" containerName="revoke-keys"
Oct 30 19:49:08.920711 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.920692    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:08.976649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:08.976617    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"]
Oct 30 19:49:09.005428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.005383    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.005632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.005497    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.005632 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.005577    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhj4\" (UniqueName: \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.017235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.017210    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:49:09.017351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.017261    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:49:09.106430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.106392    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhj4\" (UniqueName: \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.106628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.106591    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.106730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.106714    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.107368 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.107340    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.108822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.108806    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.165424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.165393    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhj4\" (UniqueName: \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\") pod \"maas-api-revoke-sub-d9b7f942-g8qdp\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.234794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.234761    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:09.612775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.612590    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"]
Oct 30 19:49:09.612898 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:49:09.612785    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86c2ea5_69a4_4358_9466_200cd3420f1f.slice/crio-86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858 WatchSource:0}: Error finding container 86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858: Status 404 returned error can't find the container with id 86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858
Oct 30 19:49:09.917637 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.917603    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp" event={"ID":"b86c2ea5-69a4-4358-9466-200cd3420f1f","Type":"ContainerStarted","Data":"1ff0a578b2ca3fa341351d0485dde681a12590256ea888ea3e913a57cd924ecd"}
Oct 30 19:49:09.917989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:09.917656    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp" event={"ID":"b86c2ea5-69a4-4358-9466-200cd3420f1f","Type":"ContainerStarted","Data":"86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858"}
Oct 30 19:49:10.029653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:10.029597    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp" podStartSLOduration=2.029582365 podStartE2EDuration="2.029582365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:10.02779728 +0000 UTC m=+1459.532201028" watchObservedRunningTime="2026-10-30 19:49:10.029582365 +0000 UTC m=+1459.533986106"
Oct 30 19:49:10.926473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:10.926442    5085 generic.go:356] "Generic (PLEG): container finished" podID="b86c2ea5-69a4-4358-9466-200cd3420f1f" containerID="1ff0a578b2ca3fa341351d0485dde681a12590256ea888ea3e913a57cd924ecd" exitCode=0
Oct 30 19:49:10.926870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:10.926527    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp" event={"ID":"b86c2ea5-69a4-4358-9466-200cd3420f1f","Type":"ContainerDied","Data":"1ff0a578b2ca3fa341351d0485dde681a12590256ea888ea3e913a57cd924ecd"}
Oct 30 19:49:12.098206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.098179    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:12.246954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.246862    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\") pod \"b86c2ea5-69a4-4358-9466-200cd3420f1f\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") "
Oct 30 19:49:12.247124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.246999    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\") pod \"b86c2ea5-69a4-4358-9466-200cd3420f1f\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") "
Oct 30 19:49:12.247192 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.247163    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\" (UniqueName: \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\") pod \"b86c2ea5-69a4-4358-9466-200cd3420f1f\" (UID: \"b86c2ea5-69a4-4358-9466-200cd3420f1f\") "
Oct 30 19:49:12.247256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.247226    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca" pod "b86c2ea5-69a4-4358-9466-200cd3420f1f" (UID: "b86c2ea5-69a4-4358-9466-200cd3420f1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:12.247628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.247608    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:12.249147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.249115    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca" pod "b86c2ea5-69a4-4358-9466-200cd3420f1f" (UID: "b86c2ea5-69a4-4358-9466-200cd3420f1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:49:12.249251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.249185    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4" pod "b86c2ea5-69a4-4358-9466-200cd3420f1f" (UID: "b86c2ea5-69a4-4358-9466-200cd3420f1f"). InnerVolumeSpecName "kube-api-access-2qhj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:12.348660 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.348628    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b86c2ea5-69a4-4358-9466-200cd3420f1f-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:12.348660 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.348654    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qhj4\" (UniqueName: \"kubernetes.io/projected/b86c2ea5-69a4-4358-9466-200cd3420f1f-kube-api-access-2qhj4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:49:12.943746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.943717    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"
Oct 30 19:49:12.943951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.943760    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp" event={"ID":"b86c2ea5-69a4-4358-9466-200cd3420f1f","Type":"ContainerDied","Data":"86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858"}
Oct 30 19:49:12.943951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:12.943809    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86166fb7544845eeaa6dd45ae28783fad0ca80e9764bb11803a3ba6c1f94d858"
Oct 30 19:49:21.939830 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:49:21.939787    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:49:26.081347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:26.081309    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:49:28.135640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.132694    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:28.239586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.238721    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:28.361587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.356660    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:28.406169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.406063    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:28.406792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.406760    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 19:49:28.465679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.464221    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:28.706352 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:28.705655    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 30 19:49:29.133775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.133655    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:29.134767 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.134743    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:49:29.248708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.248664    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:29.249695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.249671    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:49:29.423643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.418338    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:49:29.423643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.419073    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:49:29.423643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.420404    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:29.508588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.504764    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:49:29.508588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.506227    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:29.513809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.513665    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:49:29.609928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.609787    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:49:29.609928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.609853    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:29.706036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.705928    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:29.786677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:29.757651    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:49:30.081493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.081383    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:30.141631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.136803    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": context deadline exceeded"
Oct 30 19:49:30.305592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.301696    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:30.502615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.430816    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:30.502615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.431199    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:30.525488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.523656    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": context deadline exceeded"
Oct 30 19:49:30.728459 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:30.723131    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:31.241735 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.241673    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:31.243668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.243613    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:31.476746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.473861    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 30 19:49:31.520059 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.519961    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded"
Oct 30 19:49:31.546412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.546277    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:31.707980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:31.679906    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.132611 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:49:32.126071    5085 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.427s"
Oct 30 19:49:32.273587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.264128    5085 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.319684 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.319649    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:49:32.319970 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.319940    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.321511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.321483    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.335693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.335550    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.122:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.479586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.474686    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.521846 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.521808    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.548687 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.548641    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th" podUID="00a40e56-bd58-4960-98ab-e6f7be89e798" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:32.747788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.747704    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th"
Oct 30 19:49:32.748045 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.748028    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 19:49:32.748186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:32.748173    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 19:49:33.346777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:33.346411    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 19:49:33.357132 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:33.357104    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 19:49:33.380445 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:33.380404    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:49:33.445598 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:33.444615    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 19:49:48.780540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.780508    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:49:48.781139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.781125    5085 cpu_manager.go:414] "containerMap: removing container" podUID="b86c2ea5-69a4-4358-9466-200cd3420f1f" containerName="revoke-keys"
Oct 30 19:49:48.781187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.781140    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86c2ea5-69a4-4358-9466-200cd3420f1f" containerName="revoke-keys"
Oct 30 19:49:48.781256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.781247    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86c2ea5-69a4-4358-9466-200cd3420f1f" containerName="revoke-keys"
Oct 30 19:49:48.785119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.785102    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.841335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.841299    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8d3b6f-openshift-default-dockercfg-dqhs4\""
Oct 30 19:49:48.850281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850242    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.850579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850547    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.850811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850788    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.850920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850816    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.850920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850835    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.851016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850919    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.851016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.850994    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2vc\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.851100 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.851027    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.851100 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.851062    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.894875 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.894835    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:49:48.952677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952632    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952688    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952721    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952763    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952800    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2vc\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952841    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.952917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952882    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.953296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.952976    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.953296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.953023    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.953507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.953480    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.953628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.953602    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.953826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.953805    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.954942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.954807    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.954942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.953785    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.956128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.956074    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:48.982589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:48.981781    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:49.090681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.085633    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:49.098588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.092749    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2vc\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\") pod \"e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") " pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:49.399279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.399204    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:49.827679 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:49:49.827639    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0668d04_5116_4911_9668_763e3b6e0e55.slice/crio-8c8311e414729c9255f9ba6659cd58e00e9d6e68a3d15e21e10437131d5a8040 WatchSource:0}: Error finding container 8c8311e414729c9255f9ba6659cd58e00e9d6e68a3d15e21e10437131d5a8040: Status 404 returned error can't find the container with id 8c8311e414729c9255f9ba6659cd58e00e9d6e68a3d15e21e10437131d5a8040
Oct 30 19:49:49.828023 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.827936    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:49:49.829993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.829962    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:49:49.830085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.830025    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:49:49.830085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:49.830056    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:49:50.540058 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:50.540023    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerStarted","Data":"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"}
Oct 30 19:49:50.540262 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:50.540084    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerStarted","Data":"8c8311e414729c9255f9ba6659cd58e00e9d6e68a3d15e21e10437131d5a8040"}
Oct 30 19:49:50.704044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:50.703995    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podStartSLOduration=2.703981091 podStartE2EDuration="2.703981091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:50.701038687 +0000 UTC m=+1500.205442437" watchObservedRunningTime="2026-10-30 19:49:50.703981091 +0000 UTC m=+1500.208384837"
Oct 30 19:49:50.855652 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:50.855518    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:49:50.886753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:50.886723    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:49:51.215085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.215055    5085 scope.go:122] "RemoveContainer" containerID="68adcf469c80d24599286542d65d2630c1f03a0c185c43c815612baf4169c0ea"
Oct 30 19:49:51.225739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.225717    5085 scope.go:122] "RemoveContainer" containerID="f921f4944a6aa445ccc65da2531999795496850d61ae075bbf03197b4c27365f"
Oct 30 19:49:51.234931 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.234911    5085 scope.go:122] "RemoveContainer" containerID="614fd194334fa07ef6f510e4a2141b3a1790b2f1bb61e0dd79b7ae0dae3b52ab"
Oct 30 19:49:51.244352 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.244325    5085 scope.go:122] "RemoveContainer" containerID="ffcdff32950f78a83119feba4a9b4a4f4cda6f154bc0cd830902f36f3bf5a0c5"
Oct 30 19:49:51.253996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.253972    5085 scope.go:122] "RemoveContainer" containerID="9d00d6393e756093ed34e0f274db77f773e54fc0ff0f99cbaf1da1a971324b55"
Oct 30 19:49:51.399404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.399368    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:51.404556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.404531    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:51.550260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.550175    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:51.551043 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:49:51.551022    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:49:51.943175 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:49:51.943133    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:50:21.941215 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:50:21.941177    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:50:26.145194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:26.145159    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:50:29.786786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.786748    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:50:29.791461 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.791440    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.887586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.862119    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.940999 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.940968    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.941369 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.941349    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.941636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.941618    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.941820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.941805    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.941962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.941948    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.942122 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.942108    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.942289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.942272    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdd8x\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.942439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.942425    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:29.942679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.940465    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cf283d-openshift-default-dockercfg-l2hvl\""
Oct 30 19:50:29.960663 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:29.960621    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:50:30.044081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.043984    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044040    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044119    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044198    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044243    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044272    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044330 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044320    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044373    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdd8x\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044414    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.044885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.044860    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.045121 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.045094    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.045283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.045262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.045727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.045699    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.046721 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.046693    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.047508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.047482    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.047776 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.047754    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.103317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.103279    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.103518 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.103495    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdd8x\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\") pod \"e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") " pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.106205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.106182    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:30.578852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.578777    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:50:30.582203 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:50:30.582175    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb5ea7_61af_488a_b80b_89c213321588.slice/crio-244444a90041246ef303b439fceec2802eb0a1ea897afb2af671cb5fd8ccbc64 WatchSource:0}: Error finding container 244444a90041246ef303b439fceec2802eb0a1ea897afb2af671cb5fd8ccbc64: Status 404 returned error can't find the container with id 244444a90041246ef303b439fceec2802eb0a1ea897afb2af671cb5fd8ccbc64
Oct 30 19:50:30.584739 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.584706    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:30.584832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.584775    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:30.584832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.584806    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:30.958185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.958141    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerStarted","Data":"422032546a36ced680b596c5855a8a8559ee130ed6382da4e1c184831d7605aa"}
Oct 30 19:50:30.958642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:30.958214    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerStarted","Data":"244444a90041246ef303b439fceec2802eb0a1ea897afb2af671cb5fd8ccbc64"}
Oct 30 19:50:31.078722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:31.078672    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podStartSLOduration=2.078655077 podStartE2EDuration="2.078655077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:31.073539565 +0000 UTC m=+1540.577943332" watchObservedRunningTime="2026-10-30 19:50:31.078655077 +0000 UTC m=+1540.583058828"
Oct 30 19:50:31.107441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:31.107410    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:31.112755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:31.112733    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:31.972611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:31.969812    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:31.972611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:31.970828    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:50:38.712054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.712022    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:50:38.716819 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.716796    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.718637 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.718608    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:50:38.819131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.819100    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srff2\" (UniqueName: \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.819313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.819211    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.843863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.843839    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cf283d\""
Oct 30 19:50:38.843863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.843842    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cf283d-dockercfg-4wtl6\""
Oct 30 19:50:38.920441 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.920409    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.920690 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.920669    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srff2\" (UniqueName: \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.921155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.921138    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:38.980183 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:38.980112    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srff2\" (UniqueName: \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\") pod \"payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:39.032942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:39.032906    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:39.412300 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:50:39.412274    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373e9c93_69a8_48ac_b3f3_b92da7d437f0.slice/crio-cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64 WatchSource:0}: Error finding container cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64: Status 404 returned error can't find the container with id cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64
Oct 30 19:50:39.412813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:39.412789    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:50:40.035001 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.034965    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" event={"ID":"373e9c93-69a8-48ac-b3f3-b92da7d437f0","Type":"ContainerStarted","Data":"6aa98a5ed33cf867973b6287613071143bce282e195921f9400f70874e043e7f"}
Oct 30 19:50:40.035400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.035025    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" event={"ID":"373e9c93-69a8-48ac-b3f3-b92da7d437f0","Type":"ContainerStarted","Data":"cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64"}
Oct 30 19:50:40.035400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.035053    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:40.149022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.148971    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" podStartSLOduration=2.148956694 podStartE2EDuration="2.148956694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:40.145794479 +0000 UTC m=+1549.650198238" watchObservedRunningTime="2026-10-30 19:50:40.148956694 +0000 UTC m=+1549.653360440"
Oct 30 19:50:40.640784 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.640756    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"]
Oct 30 19:50:40.644938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.644916    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.699175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.699144    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"]
Oct 30 19:50:40.740658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.740630    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:50:40.740658 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.740641    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:50:40.747190 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.747164    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.747325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.747207    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.747325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.747297    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgjk\" (UniqueName: \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.848917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.848883    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.849088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.848941    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.849088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.848987    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgjk\" (UniqueName: \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.849643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.849619    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.851271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.851255    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.908144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.908064    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgjk\" (UniqueName: \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\") pod \"maas-api-revoke-sub-efa160ed-c6cbp\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:40.958996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:40.958959    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:41.335130 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:50:41.335077    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50247cbc_9e0b_45f0_aa15_6b5e4c0b1cee.slice/crio-81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f WatchSource:0}: Error finding container 81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f: Status 404 returned error can't find the container with id 81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f
Oct 30 19:50:41.336196 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:41.336170    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"]
Oct 30 19:50:42.054027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.053987    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp" event={"ID":"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee","Type":"ContainerStarted","Data":"331bf0e57448de8594d7f287c8d2392055f25db517ffda63dc62416a38be1b13"}
Oct 30 19:50:42.054227 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.054051    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp" event={"ID":"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee","Type":"ContainerStarted","Data":"81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f"}
Oct 30 19:50:42.164889 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.164836    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp" podStartSLOduration=2.164821314 podStartE2EDuration="2.164821314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:42.164015164 +0000 UTC m=+1551.668418911" watchObservedRunningTime="2026-10-30 19:50:42.164821314 +0000 UTC m=+1551.669225061"
Oct 30 19:50:42.215499 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.215466    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"]
Oct 30 19:50:42.219922 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.219907    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.276756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.276588    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"]
Oct 30 19:50:42.375318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.375284    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.375318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.375321    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229xd\" (UniqueName: \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.375779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.375489    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.476978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.476941    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.476978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.476990    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229xd\" (UniqueName: \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.477236 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.477143    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.477971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.477939    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.479503 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.479474    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.536234 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.536204    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-229xd\" (UniqueName: \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\") pod \"maas-api-revoke-sub-30bb01c0-k46gn\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:42.834391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:42.834356    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:43.063627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:43.063589    5085 generic.go:356] "Generic (PLEG): container finished" podID="50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" containerID="331bf0e57448de8594d7f287c8d2392055f25db517ffda63dc62416a38be1b13" exitCode=0
Oct 30 19:50:43.063808 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:43.063625    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp" event={"ID":"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee","Type":"ContainerDied","Data":"331bf0e57448de8594d7f287c8d2392055f25db517ffda63dc62416a38be1b13"}
Oct 30 19:50:43.210364 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:50:43.210336    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e7eea7_7bcc_4066_9518_894367d4190f.slice/crio-f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229 WatchSource:0}: Error finding container f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229: Status 404 returned error can't find the container with id f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229
Oct 30 19:50:43.229373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:43.229347    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"]
Oct 30 19:50:44.075146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.075118    5085 generic.go:356] "Generic (PLEG): container finished" podID="27e7eea7-7bcc-4066-9518-894367d4190f" containerID="daed6e5c3015d457785c7f822c2509d41e7e417cf58f1be235e408c225d930c4" exitCode=0
Oct 30 19:50:44.075457 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.075192    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn" event={"ID":"27e7eea7-7bcc-4066-9518-894367d4190f","Type":"ContainerDied","Data":"daed6e5c3015d457785c7f822c2509d41e7e417cf58f1be235e408c225d930c4"}
Oct 30 19:50:44.075457 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.075243    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn" event={"ID":"27e7eea7-7bcc-4066-9518-894367d4190f","Type":"ContainerStarted","Data":"f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229"}
Oct 30 19:50:44.231055 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.231023    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:44.402166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.402070    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\") pod \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") "
Oct 30 19:50:44.402346 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.402208    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\") pod \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") "
Oct 30 19:50:44.402346 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.402261    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\" (UniqueName: \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\") pod \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\" (UID: \"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee\") "
Oct 30 19:50:44.402466 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.402446    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca" pod "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" (UID: "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:44.402779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.402759    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:44.404315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.404289    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca" pod "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" (UID: "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:44.404395 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.404357    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk" pod "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" (UID: "50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee"). InnerVolumeSpecName "kube-api-access-hpgjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:44.504363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.504329    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:44.504363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:44.504359    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgjk\" (UniqueName: \"kubernetes.io/projected/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee-kube-api-access-hpgjk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:45.084336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.084303    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"
Oct 30 19:50:45.084845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.084337    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp" event={"ID":"50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee","Type":"ContainerDied","Data":"81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f"}
Oct 30 19:50:45.084845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.084413    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81be5bcdf91eb19fba33aa9ab9e7b1f58597a8eb1fbdfe4700eca5321022517f"
Oct 30 19:50:45.272919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.272895    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:45.416593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.416488    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\") pod \"27e7eea7-7bcc-4066-9518-894367d4190f\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") "
Oct 30 19:50:45.416753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.416664    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\") pod \"27e7eea7-7bcc-4066-9518-894367d4190f\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") "
Oct 30 19:50:45.416753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.416726    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\" (UniqueName: \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\") pod \"27e7eea7-7bcc-4066-9518-894367d4190f\" (UID: \"27e7eea7-7bcc-4066-9518-894367d4190f\") "
Oct 30 19:50:45.416944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.416916    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca" pod "27e7eea7-7bcc-4066-9518-894367d4190f" (UID: "27e7eea7-7bcc-4066-9518-894367d4190f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:45.417549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.417521    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:45.419275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.419247    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca" pod "27e7eea7-7bcc-4066-9518-894367d4190f" (UID: "27e7eea7-7bcc-4066-9518-894367d4190f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:50:45.419275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.419260    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd" pod "27e7eea7-7bcc-4066-9518-894367d4190f" (UID: "27e7eea7-7bcc-4066-9518-894367d4190f"). InnerVolumeSpecName "kube-api-access-229xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:45.518859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.518829    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e7eea7-7bcc-4066-9518-894367d4190f-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:45.518859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:45.518858    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229xd\" (UniqueName: \"kubernetes.io/projected/27e7eea7-7bcc-4066-9518-894367d4190f-kube-api-access-229xd\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:50:46.093365 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:46.093331    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn" event={"ID":"27e7eea7-7bcc-4066-9518-894367d4190f","Type":"ContainerDied","Data":"f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229"}
Oct 30 19:50:46.093365 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:46.093355    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"
Oct 30 19:50:46.093800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:46.093379    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f04d0148050fbb013d4b1a1d0894bd90c546079c3f643d7b22269f0ce92be229"
Oct 30 19:50:51.048471 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:51.048444    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:50:51.313483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:51.313386    5085 scope.go:122] "RemoveContainer" containerID="e71dabfc9fe4eb4fb6f1351a5f3037b0b81c175244969d46878d999e69bbae20"
Oct 30 19:50:51.939695 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:50:51.939663    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:50:55.292542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.292502    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 19:50:55.293440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293423    5085 cpu_manager.go:414] "containerMap: removing container" podUID="50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" containerName="revoke-keys"
Oct 30 19:50:55.293512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293443    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" containerName="revoke-keys"
Oct 30 19:50:55.293512 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293504    5085 cpu_manager.go:414] "containerMap: removing container" podUID="27e7eea7-7bcc-4066-9518-894367d4190f" containerName="revoke-keys"
Oct 30 19:50:55.293613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293512    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e7eea7-7bcc-4066-9518-894367d4190f" containerName="revoke-keys"
Oct 30 19:50:55.293666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293653    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e7eea7-7bcc-4066-9518-894367d4190f" containerName="revoke-keys"
Oct 30 19:50:55.293707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.293674    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" containerName="revoke-keys"
Oct 30 19:50:55.299706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.299684    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.354534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.353471    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-111f9c64-openshift-default-dockercfg-dn49t\""
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.521989    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522045    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8c6n\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522103    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522151    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522204    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522267    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522396    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522437    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.528664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.522521    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624519    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624594    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624671    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624818    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624845    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8c6n\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624892    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624934    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.624986    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.625043    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.626145    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.626475    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.626733    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.628438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.628270    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.629377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.628498    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.629377 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.629320    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.632465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.632387    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.635707 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.635685    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 19:50:55.705027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.704903    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8c6n\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.712242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.705952    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\") pod \"e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") " pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:55.927237 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:55.927201    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:56.377069 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:50:56.377033    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91e1afb_7f26_45fe_bc8d_0c3b8bc41390.slice/crio-57c2262cbfa4c962d4c2fbcc7c0794d22540c0e32d3ae8d85db0287bf6d034fd WatchSource:0}: Error finding container 57c2262cbfa4c962d4c2fbcc7c0794d22540c0e32d3ae8d85db0287bf6d034fd: Status 404 returned error can't find the container with id 57c2262cbfa4c962d4c2fbcc7c0794d22540c0e32d3ae8d85db0287bf6d034fd
Oct 30 19:50:56.378852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:56.378785    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 19:50:56.379706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:56.379674    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:56.379812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:56.379750    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:56.379878 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:56.379808    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:50:57.220791 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:57.220755    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerStarted","Data":"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"}
Oct 30 19:50:57.221004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:57.220820    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerStarted","Data":"57c2262cbfa4c962d4c2fbcc7c0794d22540c0e32d3ae8d85db0287bf6d034fd"}
Oct 30 19:50:57.385177 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:57.385128    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podStartSLOduration=2.385113672 podStartE2EDuration="2.385113672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:57.382754305 +0000 UTC m=+1566.887158066" watchObservedRunningTime="2026-10-30 19:50:57.385113672 +0000 UTC m=+1566.889517418"
Oct 30 19:50:57.928373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:57.928276    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:57.934403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:57.934377    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:58.230541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:58.230461    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:50:58.231515 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:50:58.231495    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:51:01.266392 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.266351    5085 generic.go:356] "Generic (PLEG): container finished" podID="b0668d04-5116-4911-9668-763e3b6e0e55" containerID="7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c" exitCode=0
Oct 30 19:51:01.266837 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.266421    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerDied","Data":"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"}
Oct 30 19:51:01.266892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.266840    5085 scope.go:122] "RemoveContainer" containerID="7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"
Oct 30 19:51:01.268244 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.268212    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:01.268355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.268292    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:01.268355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:01.268338    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:02.297770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:02.297725    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerStarted","Data":"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"}
Oct 30 19:51:02.399657 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:02.399624    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:51:02.400897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:02.400869    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:02.401031 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:02.400916    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:03.399514 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:03.399477    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:03.399932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:03.399521    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:04.457678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.455748    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:04.457678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.455822    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:04.511238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.511202    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"]
Oct 30 19:51:04.521594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.521534    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.523679 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.523651    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"]
Oct 30 19:51:04.596582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.596539    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:51:04.607345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.607315    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.607483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.607363    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlkk\" (UniqueName: \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.607595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.607551    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.627668 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.627643    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:51:04.709054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.708965    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.709309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.709276    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlkk\" (UniqueName: \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.709510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.709488    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.710185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.710166    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.711803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.711775    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.786642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.786596    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlkk\" (UniqueName: \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\") pod \"maas-api-revoke-sub-a66c7306-fgq47\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:04.841282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:04.841245    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:05.248116 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:51:05.248074    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce61b513_c882_4c3e_8b84_5f2cbd3c3ac0.slice/crio-560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3 WatchSource:0}: Error finding container 560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3: Status 404 returned error can't find the container with id 560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3
Oct 30 19:51:05.248483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:05.248457    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"]
Oct 30 19:51:05.332443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:05.332406    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47" event={"ID":"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0","Type":"ContainerStarted","Data":"560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3"}
Oct 30 19:51:05.400175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:05.400136    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:05.400342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:05.400221    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:06.342124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:06.342089    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47" event={"ID":"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0","Type":"ContainerStarted","Data":"79ec794eca429a29359ca3f59982cb8c059393a28af42e340bbe3fbb18201c22"}
Oct 30 19:51:06.400754 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:06.400727    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:06.400883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:06.400781    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:06.454973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:06.454922    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47" podStartSLOduration=2.454899389 podStartE2EDuration="2.454899389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:06.452492709 +0000 UTC m=+1575.956896467" watchObservedRunningTime="2026-10-30 19:51:06.454899389 +0000 UTC m=+1575.959303137"
Oct 30 19:51:06.550202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:06.550171    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:51:07.352140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:07.352103    5085 generic.go:356] "Generic (PLEG): container finished" podID="ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" containerID="79ec794eca429a29359ca3f59982cb8c059393a28af42e340bbe3fbb18201c22" exitCode=0
Oct 30 19:51:07.352540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:07.352168    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47" event={"ID":"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0","Type":"ContainerDied","Data":"79ec794eca429a29359ca3f59982cb8c059393a28af42e340bbe3fbb18201c22"}
Oct 30 19:51:07.400336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:07.400302    5085 patch_prober.go:29] interesting pod/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body=
Oct 30 19:51:07.400486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:07.400354    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused"
Oct 30 19:51:08.405223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.405193    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:51:08.406363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.406339    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:51:08.513989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.513966    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:08.665748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.665668    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\" (UniqueName: \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\") pod \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") "
Oct 30 19:51:08.665917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.665827    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\") pod \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") "
Oct 30 19:51:08.666006 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.665992    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\") pod \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\" (UID: \"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0\") "
Oct 30 19:51:08.666300 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.666280    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca" pod "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" (UID: "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:08.666468 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.666451    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:08.668304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.668280    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca" pod "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" (UID: "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:51:08.668410 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.668333    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk" pod "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" (UID: "ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0"). InnerVolumeSpecName "kube-api-access-tvlkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:08.768687 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.768651    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:08.768687 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:08.768689    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlkk\" (UniqueName: \"kubernetes.io/projected/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0-kube-api-access-tvlkk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:51:09.369537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:09.369504    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"
Oct 30 19:51:09.369726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:09.369533    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47" event={"ID":"ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0","Type":"ContainerDied","Data":"560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3"}
Oct 30 19:51:09.369726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:09.369622    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="560acc9fd8a9d7533c7f7af9c4c8548297550f8c5fff7aff0c615824d66389a3"
Oct 30 19:51:10.981856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:10.981823    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:51:11.291966 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.291887    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:51:11.292573 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.292545    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" containerName="revoke-keys"
Oct 30 19:51:11.292630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.292576    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" containerName="revoke-keys"
Oct 30 19:51:11.292704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.292695    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" containerName="revoke-keys"
Oct 30 19:51:11.296627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.296610    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.384932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.384908    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4477e2-openshift-default-dockercfg-jxr4f\""
Oct 30 19:51:11.399880 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.399848    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:51:11.405280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405245    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405474 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405309    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405474 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405439    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405558    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405655    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405785 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405771    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.405874 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405845    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lq9\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.406410 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405899    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.406410 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.405928    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.539586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.515463    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.546766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.546679    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.569745    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.569824    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lq9\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.569868    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.569900    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.570117    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.570177    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.570216    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.570743    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.516044    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.572587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.571445    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.577590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.577209    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.577590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.577514    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.584588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.580622    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.584791 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.582160    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.639617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.639516    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.652282 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.652243    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lq9\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\") pod \"e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") " pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:11.911411 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:11.911328    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:12.539496 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:51:12.539467    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af2d63d_f72b_4cf3_904d_b158b53c7167.slice/crio-4430fa3983ba93b4ea178a944ee217919a7ad781643d37e55747b758bd78613f WatchSource:0}: Error finding container 4430fa3983ba93b4ea178a944ee217919a7ad781643d37e55747b758bd78613f: Status 404 returned error can't find the container with id 4430fa3983ba93b4ea178a944ee217919a7ad781643d37e55747b758bd78613f
Oct 30 19:51:12.540280 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:12.540254    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:51:12.543194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:12.543155    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:12.543403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:12.543377    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:12.543557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:12.543536    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:13.439588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:13.427536    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerStarted","Data":"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"}
Oct 30 19:51:13.439588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:13.427674    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerStarted","Data":"4430fa3983ba93b4ea178a944ee217919a7ad781643d37e55747b758bd78613f"}
Oct 30 19:51:13.615726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:13.601833    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podStartSLOduration=2.6018156660000002 podStartE2EDuration="2.601815666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:13.601650411 +0000 UTC m=+1583.106054171" watchObservedRunningTime="2026-10-30 19:51:13.601815666 +0000 UTC m=+1583.106219414"
Oct 30 19:51:13.924636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:13.916699    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:13.925078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:13.925051    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:14.748586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:14.745671    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:14.748586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:14.745750    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:14.808588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:14.800730    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:14.808588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:14.800824    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:15.017734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:15.017640    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:15.025586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:15.018652    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:15.038792 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:15.038762    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:15.039098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:15.039078    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:51:15.332753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:15.332654    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:16.259902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.259863    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:51:16.267443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.267410    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.320255 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.320215    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-61d11a-openshift-default-dockercfg-fcbxs\""
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410556    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410679    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410724    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfw95\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410790    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410820    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410911    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410933    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.410993    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.421583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.411066    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.515615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.515437    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:51:16.516221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.516191    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.516539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.516503    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.516861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.516837    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517007    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfw95\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517168    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517307 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517292    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517478 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517460    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517589    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.517790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.517769    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.519113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.519091    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.519323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.519295    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.519907 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.519873    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.520610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.520586    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.520912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.520888    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.521284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.521267    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.527582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.526928    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.588331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.588293    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfw95\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.595091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.595060    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\") pod \"e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") " pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:16.883547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:16.883474    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:17.332391 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:17.332358    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:51:17.333387 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:51:17.333360    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321f5c68_f4f7_4c12_87bb_d10f29ed3517.slice/crio-e0168c317827a75fd12397cc845fa991d70470ec30a2ca2c3e9227dbb29244c5 WatchSource:0}: Error finding container e0168c317827a75fd12397cc845fa991d70470ec30a2ca2c3e9227dbb29244c5: Status 404 returned error can't find the container with id e0168c317827a75fd12397cc845fa991d70470ec30a2ca2c3e9227dbb29244c5
Oct 30 19:51:17.336277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:17.336243    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:17.336382 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:17.336305    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:17.336382 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:17.336333    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:18.072204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.071158    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerStarted","Data":"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"}
Oct 30 19:51:18.072204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.071219    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerStarted","Data":"e0168c317827a75fd12397cc845fa991d70470ec30a2ca2c3e9227dbb29244c5"}
Oct 30 19:51:18.194508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.194454    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podStartSLOduration=2.194441603 podStartE2EDuration="2.194441603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:18.188514878 +0000 UTC m=+1587.692918626" watchObservedRunningTime="2026-10-30 19:51:18.194441603 +0000 UTC m=+1587.698845350"
Oct 30 19:51:18.291539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.291502    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:51:18.296112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.296093    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.358204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.356643    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.374917    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.374959    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7dz\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375079    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375115    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375141    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375180    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375226    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375252    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.375493 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.375279    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.476764 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.476734    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.476947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.476769    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7dz\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.476947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.476926    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.476962    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.476992    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477026    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477074    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477107    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477148    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477487    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477552 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477490    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477652    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477712    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.477794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.477759    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.479321 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.479296    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.479900 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.479882    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.535202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.535173    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7dz\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.535202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.535192    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\") pod \"e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") " pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.615285 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.615206    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:18.884054 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.883971    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:18.889508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:18.889484    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:19.069607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.066731    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:51:19.073916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.073868    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:19.074040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.074010    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:19.074118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.074095    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:19.084171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.084135    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" event={"ID":"43905552-d8e5-43a8-a012-062a18155336","Type":"ContainerStarted","Data":"54706f485866b43e0d5d3166e27e0a426a7c8955c2ec9a80cc188ce4eea33de0"}
Oct 30 19:51:19.084451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.084435    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:19.085638 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:19.085618    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:51:20.096625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:20.096555    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" event={"ID":"43905552-d8e5-43a8-a012-062a18155336","Type":"ContainerStarted","Data":"4eeaf6609b8daf8b4c12bf15e3797ec31e4a0b0164a5e27cf7a3a4ad57467ac3"}
Oct 30 19:51:20.416645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:20.410297    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podStartSLOduration=2.410278591 podStartE2EDuration="2.410278591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:20.409954023 +0000 UTC m=+1589.914357770" watchObservedRunningTime="2026-10-30 19:51:20.410278591 +0000 UTC m=+1589.914682338"
Oct 30 19:51:20.617611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:20.617363    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:20.624593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:20.621777    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:20.624593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:20.621848    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:21.615950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:21.615917    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:21.616337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:21.615977    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:21.939743 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:51:21.939706    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:51:22.617221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:22.617079    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:22.617950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:22.617294    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:23.615787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:23.615749    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:23.615954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:23.615820    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:24.615959 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:24.615922    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:24.616360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:24.615989    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:25.615725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:25.615690    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 19:51:25.615958 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:25.615748    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 19:51:26.714520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:26.714483    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:26.715776 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:26.715727    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:26.717997 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:26.717924    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:51:31.889586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:31.888662    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-mcbsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:51:31.889586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:31.888747    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-mcbsf" podUID="f7eff08b-642e-4125-90e1-88ea8adf5385" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:32.510606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:32.510266    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:32.535582 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:32.534557    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:51:42.258691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.258652    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 19:51:42.263442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.263421    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.313430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.313394    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:51:42.313430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.313400    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:51:42.313672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.313431    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:51:42.313672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.313483    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jkf5r\""
Oct 30 19:51:42.366553 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.366519    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 19:51:42.376546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.376517    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.376546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.376547    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.376829 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.376594    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.376916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.376891    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.376971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.376930    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpz8\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.377085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.377067    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.377150 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.377117    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478391    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478483    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478511    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478650 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478548    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478773    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478797    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpz8\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.478938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.478888    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.479577 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.479538    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.481225 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.481189    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.481332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.481222    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.481472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.481448    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.481696 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.481678    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.538205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.538137    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.538384 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.538363    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpz8\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-mj77z\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:42.578541 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:42.578512    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:43.006488 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:51:43.006452    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015a4e2e_3779_458c_b7d4_74c6a149e906.slice/crio-843a6067f1e248d3cdb4fd55ca6d62988b3f44220b034fd7963cb48821b06ad8 WatchSource:0}: Error finding container 843a6067f1e248d3cdb4fd55ca6d62988b3f44220b034fd7963cb48821b06ad8: Status 404 returned error can't find the container with id 843a6067f1e248d3cdb4fd55ca6d62988b3f44220b034fd7963cb48821b06ad8
Oct 30 19:51:43.006675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:43.006652    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 19:51:43.008722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:43.008702    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:51:43.472647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:43.472613    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" event={"ID":"015a4e2e-3779-458c-b7d4-74c6a149e906","Type":"ContainerStarted","Data":"843a6067f1e248d3cdb4fd55ca6d62988b3f44220b034fd7963cb48821b06ad8"}
Oct 30 19:51:47.132974 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:47.132937    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:47.133314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:47.133003    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:51:47.511113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:47.511072    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" event={"ID":"015a4e2e-3779-458c-b7d4-74c6a149e906","Type":"ContainerStarted","Data":"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"}
Oct 30 19:51:47.511294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:47.511141    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:47.723920 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:47.723864    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" podStartSLOduration=1.601079345 podStartE2EDuration="5.72384992s" totalImagesPullingTime="4.122770575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:47.673166684 +0000 UTC m=+1617.177570431" watchObservedRunningTime="2026-10-30 19:51:47.72384992 +0000 UTC m=+1617.228253667"
Oct 30 19:51:48.540793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:48.540753    5085 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-mj77z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:51:48.541360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:48.541327    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:51:51.537415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:51.537387    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 19:51:51.939855 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:51:51.939823    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:51:53.692534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:53.692498    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:51:55.051557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:55.051526    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"]
Oct 30 19:51:55.097065 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:55.097033    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6856691-vtq7r"]
Oct 30 19:51:56.632387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:51:56.632351    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33315cd-a673-40be-b2ee-4fcb23ebcde8" path="/var/lib/kubelet/pods/e33315cd-a673-40be-b2ee-4fcb23ebcde8/volumes"
Oct 30 19:52:11.055379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:11.055346    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"]
Oct 30 19:52:11.107748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:11.107702    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4bb406e-xtvkk"]
Oct 30 19:52:12.634336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:12.634303    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977" path="/var/lib/kubelet/pods/5e5d372a-c5bb-4cfc-9ed8-68c0d7db9977/volumes"
Oct 30 19:52:15.054737 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:15.054647    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"]
Oct 30 19:52:15.113972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:15.113931    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a874ae99-qql84"]
Oct 30 19:52:16.633877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:16.633792    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2895eb80-d625-4bb9-b05f-89936ce19e28" path="/var/lib/kubelet/pods/2895eb80-d625-4bb9-b05f-89936ce19e28/volumes"
Oct 30 19:52:21.940960 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:52:21.940926    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.006688    5085 generic.go:356] "Generic (PLEG): container finished" podID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerID="2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8" exitCode=0
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.006758    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerDied","Data":"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"}
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.007214    5085 scope.go:122] "RemoveContainer" containerID="2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.008758    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.008845    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:22.023143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.008902    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:22.060357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.060326    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"]
Oct 30 19:52:22.065336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.065314    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-210f37af-prlwk"]
Oct 30 19:52:22.634042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:22.633968    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4" path="/var/lib/kubelet/pods/fbc6bf61-a8c3-4ffc-bd56-0f341a5605a4/volumes"
Oct 30 19:52:23.017832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.017797    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerStarted","Data":"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"}
Oct 30 19:52:23.020927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.020894    5085 generic.go:356] "Generic (PLEG): container finished" podID="f7bb5ea7-61af-488a-b80b-89c213321588" containerID="422032546a36ced680b596c5855a8a8559ee130ed6382da4e1c184831d7605aa" exitCode=0
Oct 30 19:52:23.021088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.020968    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerDied","Data":"422032546a36ced680b596c5855a8a8559ee130ed6382da4e1c184831d7605aa"}
Oct 30 19:52:23.021355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.021340    5085 scope.go:122] "RemoveContainer" containerID="422032546a36ced680b596c5855a8a8559ee130ed6382da4e1c184831d7605aa"
Oct 30 19:52:23.022885 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.022854    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:23.022995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.022932    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:23.022995 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.022964    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:23.928189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.928150    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:52:23.929476 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.929443    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 19:52:23.929643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:23.929502    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 19:52:24.034544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.034510    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerStarted","Data":"0d7c2e19eaf207a3775e0a73a0bf2acbb5ab04bf3832634d0ba64b0bdd39bfcb"}
Oct 30 19:52:24.111158 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.111117    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:52:24.112222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.112186    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:24.112438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.112412    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:24.928832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.928796    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 19:52:24.929005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:24.928855    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 19:52:25.108040 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:25.108005    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:25.108649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:25.108615    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:25.928510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:25.928472    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 19:52:25.928830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:25.928798    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 19:52:26.137946 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:26.137909    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:26.167705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:26.167654    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:26.927906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:26.927871    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 19:52:26.928115 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:26.927926    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 19:52:27.108769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:27.108721    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.109189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:27.109138    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:27.928368 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:27.928287    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.928814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:27.928354    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 19:52:28.107230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:28.107188    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.108088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:28.108044    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:28.230763 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:28.230673    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:52:28.932543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:28.932515    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:52:29.106962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:29.106929    5085 patch_prober.go:29] interesting pod/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.107126 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:29.106990    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 19:52:29.151813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:29.151788    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 19:52:30.111592 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.110619    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"]
Oct 30 19:52:30.135409 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.135376    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:52:30.137172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.137148    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:52:30.141793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.141771    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:52:30.157615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.157548    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a6e78c9-5bgdn"]
Oct 30 19:52:30.310485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.310451    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:52:30.702180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:30.702145    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdcc549-d170-42ea-9d0e-73c174d9956e" path="/var/lib/kubelet/pods/dcdcc549-d170-42ea-9d0e-73c174d9956e/volumes"
Oct 30 19:52:40.060821 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:40.060788    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"]
Oct 30 19:52:40.110765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:40.110727    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23bbe81b-gl7m5"]
Oct 30 19:52:40.633865 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:40.633831    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d" path="/var/lib/kubelet/pods/a5e03397-4b6a-4aef-bdb7-c1d4eac90f4d/volumes"
Oct 30 19:52:41.050311 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.050231    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"]
Oct 30 19:52:41.096831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.096801    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e4f1b52-74zn5"]
Oct 30 19:52:41.282425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.282390    5085 generic.go:356] "Generic (PLEG): container finished" podID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerID="2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2" exitCode=0
Oct 30 19:52:41.283021 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.282467    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerDied","Data":"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"}
Oct 30 19:52:41.283322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.283297    5085 scope.go:122] "RemoveContainer" containerID="2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"
Oct 30 19:52:41.285544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.285500    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:41.285722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.285619    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:41.285722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:41.285667    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:52:42.292534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:42.292496    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerStarted","Data":"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"}
Oct 30 19:52:42.632895 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:42.632820    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b4e163-fd71-4eaf-bbab-387c76879cc6" path="/var/lib/kubelet/pods/d3b4e163-fd71-4eaf-bbab-387c76879cc6/volumes"
Oct 30 19:52:42.883968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:42.883873    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:52:42.884150 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:42.884112    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:52:42.884230 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:42.884169    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:52:43.884406 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:43.884374    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:52:43.884811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:43.884423    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:52:44.884403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:44.884310    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:52:44.884403 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:44.884362    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:52:45.884219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:45.884176    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:52:45.884433 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:45.884239    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:52:46.050297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:46.050252    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"]
Oct 30 19:52:46.097236 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:46.097198    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-674eceb5-l9j5s"]
Oct 30 19:52:46.632867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:46.632835    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbb98ca-f0e6-4260-a2b9-c488c1159fd1" path="/var/lib/kubelet/pods/abbb98ca-f0e6-4260-a2b9-c488c1159fd1/volumes"
Oct 30 19:52:46.884888 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:46.884791    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 19:52:46.884888 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:46.884863    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 19:52:47.889212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:47.889179    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:52:47.889731 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:47.889672    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:52:47.890481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:47.890460    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:52:51.939955 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:52:51.939918    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:52:58.083906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:58.083865    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"]
Oct 30 19:52:58.103845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:58.103793    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1ad1e6-vs5zb"]
Oct 30 19:52:58.633357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:58.633323    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e19118-1d38-4faa-9543-4e17c51cad2c" path="/var/lib/kubelet/pods/d9e19118-1d38-4faa-9543-4e17c51cad2c/volumes"
Oct 30 19:52:59.052311 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:59.052278    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"]
Oct 30 19:52:59.098532 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:52:59.098501    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4893bd8-nbcpl"]
Oct 30 19:53:00.054218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:00.054183    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"]
Oct 30 19:53:00.100127 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:00.100086    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5174c56c-bj79z"]
Oct 30 19:53:00.632251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:00.632216    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1736694d-91fc-4895-8339-7754f5996c3a" path="/var/lib/kubelet/pods/1736694d-91fc-4895-8339-7754f5996c3a/volumes"
Oct 30 19:53:00.632716 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:00.632700    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce394d1-3507-4d6a-80ef-cfa36122fc07" path="/var/lib/kubelet/pods/9ce394d1-3507-4d6a-80ef-cfa36122fc07/volumes"
Oct 30 19:53:08.087273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:08.087230    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"]
Oct 30 19:53:08.129134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:08.129098    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2552377d-xlx6g"]
Oct 30 19:53:08.632268 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:08.632235    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4363df01-af0f-417e-b8b0-21f04631476a" path="/var/lib/kubelet/pods/4363df01-af0f-417e-b8b0-21f04631476a/volumes"
Oct 30 19:53:10.082641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:10.082597    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"]
Oct 30 19:53:10.107242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:10.107214    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33cf573d-c5r62"]
Oct 30 19:53:10.633780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:10.633747    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec54adc-a1a9-401b-af4b-05e7f6d86fac" path="/var/lib/kubelet/pods/fec54adc-a1a9-401b-af4b-05e7f6d86fac/volumes"
Oct 30 19:53:11.005176 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.005139    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"]
Oct 30 19:53:11.010109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.010083    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.090104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.090070    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"]
Oct 30 19:53:11.119726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.119695    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:53:11.119877 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.119759    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:53:11.148016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.147984    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.148187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.148104    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjzt\" (UniqueName: \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.148239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.148206    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.250511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.250480    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.250741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.250539    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjzt\" (UniqueName: \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.250741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.250635    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.251278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.251250    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.252930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.252910    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.309947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.309870    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjzt\" (UniqueName: \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\") pod \"maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.323672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.323639    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:11.732701 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:53:11.732663    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccc9475_115c_4fe1_b3e1_706a8d47dbd3.slice/crio-c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a WatchSource:0}: Error finding container c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a: Status 404 returned error can't find the container with id c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a
Oct 30 19:53:11.733109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.733084    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"]
Oct 30 19:53:11.925984 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:11.925951    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:53:12.652804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:12.652713    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7" event={"ID":"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3","Type":"ContainerStarted","Data":"0d771639cd5a673a5b1ddd464af424a11e7b5d851dc487d70ce7a120d28aa514"}
Oct 30 19:53:12.652804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:12.652782    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7" event={"ID":"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3","Type":"ContainerStarted","Data":"c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a"}
Oct 30 19:53:12.772704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:12.772634    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7" podStartSLOduration=2.7726148630000003 podStartE2EDuration="2.772614863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:12.768215485 +0000 UTC m=+1702.272619233" watchObservedRunningTime="2026-10-30 19:53:12.772614863 +0000 UTC m=+1702.277018612"
Oct 30 19:53:13.664774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:13.664741    5085 generic.go:356] "Generic (PLEG): container finished" podID="7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" containerID="0d771639cd5a673a5b1ddd464af424a11e7b5d851dc487d70ce7a120d28aa514" exitCode=0
Oct 30 19:53:13.665164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:13.664831    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7" event={"ID":"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3","Type":"ContainerDied","Data":"0d771639cd5a673a5b1ddd464af424a11e7b5d851dc487d70ce7a120d28aa514"}
Oct 30 19:53:14.846678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.846648    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:14.880747 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.880713    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\" (UniqueName: \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\") pod \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") "
Oct 30 19:53:14.880946 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.880925    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\") pod \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") "
Oct 30 19:53:14.881009 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.880992    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\") pod \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\" (UID: \"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3\") "
Oct 30 19:53:14.881353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.881329    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca" pod "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" (UID: "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:53:14.881544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.881521    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.883856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.883822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca" pod "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" (UID: "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:53:14.883968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.883942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt" pod "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" (UID: "7ccc9475-115c-4fe1-b3e1-706a8d47dbd3"). InnerVolumeSpecName "kube-api-access-9cjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:53:14.982905 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.982808    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjzt\" (UniqueName: \"kubernetes.io/projected/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-kube-api-access-9cjzt\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:14.982905 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:14.982844    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:53:15.681822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:15.681785    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"
Oct 30 19:53:15.681991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:15.681789    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7" event={"ID":"7ccc9475-115c-4fe1-b3e1-706a8d47dbd3","Type":"ContainerDied","Data":"c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a"}
Oct 30 19:53:15.681991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:15.681912    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f02822a244dbe45c38008770c37176264fa977e76f0c49f1bca3b89dccf50a"
Oct 30 19:53:15.994800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:15.994703    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:53:15.995298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:15.994793    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:53:16.691304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.691272    5085 generic.go:356] "Generic (PLEG): container finished" podID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerID="3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138" exitCode=0
Oct 30 19:53:16.691542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.691350    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerDied","Data":"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"}
Oct 30 19:53:16.691868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.691851    5085 scope.go:122] "RemoveContainer" containerID="3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"
Oct 30 19:53:16.693447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.693416    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:16.693543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.693497    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:16.693647 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:16.693545    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:17.057056 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.056966    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"]
Oct 30 19:53:17.061744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.061706    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f56c068-xm6hb"]
Oct 30 19:53:17.710012 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.709978    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerStarted","Data":"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"}
Oct 30 19:53:17.912004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.911977    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:53:17.912186 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.912165    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:17.912249 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:17.912231    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:18.635531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:18.635499    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705" path="/var/lib/kubelet/pods/e154bcfe-a1d0-4323-8b5a-a5c4e7e3a705/volumes"
Oct 30 19:53:18.911960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:18.911875    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:18.911960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:18.911937    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:19.912104 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:19.912067    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:19.912520 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:19.912134    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:20.912094 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:20.912055    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:20.912284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:20.912110    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:21.912129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:21.912086    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:21.912335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:21.912184    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:21.939892 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:53:21.939860    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:53:22.924275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:22.914136    5085 patch_prober.go:29] interesting pod/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 19:53:22.924275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:22.914214    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 19:53:23.916128 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:23.916081    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:53:23.916803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:23.916778    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:53:23.917664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:23.917644    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:53:25.125480 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.125437    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:53:25.126400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.126379    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" containerName="revoke-keys"
Oct 30 19:53:25.126400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.126400    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" containerName="revoke-keys"
Oct 30 19:53:25.126614 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.126598    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" containerName="revoke-keys"
Oct 30 19:53:25.130029 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.130005    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.234070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.234036    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e67696-openshift-default-dockercfg-6nnfw\""
Oct 30 19:53:25.252446 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.252387    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.253474 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.253425    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.253810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.253769    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.254007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.253978    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.254200 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.254184    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb7xh\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.254412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.254398    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.254644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.254628    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.254799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.254771    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.255049 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.255023    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.334429 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.325961    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356442    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356518    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356545    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356634    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb7xh\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356725    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356788    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356813    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356923    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.356956    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.357479    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.358220    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.358904    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.359164    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.362581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.359766    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.379025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.362888    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.379025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.374789    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.417717 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.417667    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.420036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.419992    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb7xh\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\") pod \"e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") " pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.465294 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.464789    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:25.961206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.958867    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:53:25.963845 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:53:25.963813    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51cb27e2_420d_472f_95cf_2d2ea5359485.slice/crio-87c3f4de54edc52bbbfd070005c2086c165c7a82c7390dd1a0525086b0a1dfac WatchSource:0}: Error finding container 87c3f4de54edc52bbbfd070005c2086c165c7a82c7390dd1a0525086b0a1dfac: Status 404 returned error can't find the container with id 87c3f4de54edc52bbbfd070005c2086c165c7a82c7390dd1a0525086b0a1dfac
Oct 30 19:53:25.966759 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.966716    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:25.966852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.966805    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:25.966852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:25.966842    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:53:26.785743 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:26.785706    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerStarted","Data":"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"}
Oct 30 19:53:26.786166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:26.785764    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerStarted","Data":"87c3f4de54edc52bbbfd070005c2086c165c7a82c7390dd1a0525086b0a1dfac"}
Oct 30 19:53:26.959590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:26.951626    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podStartSLOduration=1.951604465 podStartE2EDuration="1.951604465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:26.949070064 +0000 UTC m=+1716.453473840" watchObservedRunningTime="2026-10-30 19:53:26.951604465 +0000 UTC m=+1716.456008213"
Oct 30 19:53:27.465318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:27.465285    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:27.470952 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:27.470925    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:27.794195 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:27.794119    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:27.794976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:27.794958    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:53:30.078981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:30.078945    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"]
Oct 30 19:53:30.125447 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:30.125415    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccd7852e-n5pzb"]
Oct 30 19:53:30.633524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:30.633488    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebbd892-93d5-42b4-ab9c-cc944e114841" path="/var/lib/kubelet/pods/1ebbd892-93d5-42b4-ab9c-cc944e114841/volumes"
Oct 30 19:53:34.050613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:34.050581    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"]
Oct 30 19:53:34.096960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:34.096932    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d637a5e-r22r5"]
Oct 30 19:53:34.633107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:34.633076    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c303019-523f-4bd8-b451-838e39718005" path="/var/lib/kubelet/pods/2c303019-523f-4bd8-b451-838e39718005/volumes"
Oct 30 19:53:43.050083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:43.050048    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"]
Oct 30 19:53:43.096686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:43.096655    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3bc3a4a-7bvl6"]
Oct 30 19:53:44.634713 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:44.634679    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f65066-7f6e-4c76-bf09-53b4f6d61dcc" path="/var/lib/kubelet/pods/85f65066-7f6e-4c76-bf09-53b4f6d61dcc/volumes"
Oct 30 19:53:47.049108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:47.049076    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"]
Oct 30 19:53:47.095862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:47.095831    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3d35bd2-k4xpj"]
Oct 30 19:53:48.632440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:48.632409    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58ecf83-180a-4ec7-9e94-65d20827e972" path="/var/lib/kubelet/pods/e58ecf83-180a-4ec7-9e94-65d20827e972/volumes"
Oct 30 19:53:50.774279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:50.774239    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:53:51.412587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.408099    5085 scope.go:122] "RemoveContainer" containerID="411951bbfb6ebff5370cf0f4671fd0d36533e09b0520996cb71df1d882ee4b79"
Oct 30 19:53:51.458266 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.458242    5085 scope.go:122] "RemoveContainer" containerID="a2e256c1aef91e1f3bae598a50a973a8db29275aaa664832ca3942645eda0107"
Oct 30 19:53:51.492253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.491757    5085 scope.go:122] "RemoveContainer" containerID="cc4bad6cae2840cceff6024f5dce3e25b970b38328db1a39be6933e43d3c46f4"
Oct 30 19:53:51.555624 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.555590    5085 scope.go:122] "RemoveContainer" containerID="53882d9ed4d43edd10ea5cce8caef03a80e0917e175945cf5efb5a95afd4be25"
Oct 30 19:53:51.585626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.585594    5085 scope.go:122] "RemoveContainer" containerID="8c04d2ec75c014652e1fc0a7e13d5116dd84dd1d4d8ae64a3e6381de48606d79"
Oct 30 19:53:51.598510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.598487    5085 scope.go:122] "RemoveContainer" containerID="d704ef2067760f605ca8ea6afe3779cb073f39434cef84b580c98b2e05e212c0"
Oct 30 19:53:51.609914 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.609882    5085 scope.go:122] "RemoveContainer" containerID="e4fbe9f817c8a7a8eaa43f4127fc1e669e546c66595491c7b15e5c9162912c37"
Oct 30 19:53:51.622076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:53:51.622055    5085 scope.go:122] "RemoveContainer" containerID="80aeec79c660c975b27cd0289c178ffa2fbccb9eddc7433afffdfddd3231bdad"
Oct 30 19:53:51.944015 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:53:51.943974    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:54:04.100595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:04.100482    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"]
Oct 30 19:54:04.104790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:04.104765    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7ca52ec-2b65l"]
Oct 30 19:54:04.632615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:04.632552    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf" path="/var/lib/kubelet/pods/7ceb5ddc-9cc9-4c8b-8ba3-191d497b96cf/volumes"
Oct 30 19:54:13.051807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:13.051771    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"]
Oct 30 19:54:13.098336 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:13.098302    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9b7f942-g8qdp"]
Oct 30 19:54:14.638333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:14.638289    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86c2ea5-69a4-4358-9466-200cd3420f1f" path="/var/lib/kubelet/pods/b86c2ea5-69a4-4358-9466-200cd3420f1f/volumes"
Oct 30 19:54:20.877184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:20.877150    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:54:21.946166 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:54:21.946127    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:54:22.077984 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.077949    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:54:22.079686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.079653    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" containerID="cri-o://fd578ca624a089b7bffeca07adff74b8ed6a4c7a4a2338436a73857a5d7f49b7" gracePeriod=30
Oct 30 19:54:22.117673 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.117635    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:54:22.117924 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.117896    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" containerID="cri-o://53c9c163ab0a56a4ca7cc62616d1a554284682d0a36838dd7c9009d632e86bf3" gracePeriod=30
Oct 30 19:54:22.122575 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.122534    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:54:22.122761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.122738    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" containerID="cri-o://0aec34dc0f8a294d62d713822a5363d50d8a0b498ec63bbb821f886d8c80c498" gracePeriod=30
Oct 30 19:54:22.164349 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.164316    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:54:22.164580 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.164540    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" containerID="cri-o://f5c1e3493e24e44b89df656a963a0b3217eea0075119dfc767da5a1468e81c9b" gracePeriod=30
Oct 30 19:54:22.830838 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.830804    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:22.831011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:22.830859    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:23.261710 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:23.261677    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:23.262081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:23.261750    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:23.511435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:23.511398    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:23.511651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:23.511455    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:26.312661 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:26.312628    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:26.313152 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:26.312698    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:27.328774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.328587    5085 generic.go:356] "Generic (PLEG): container finished" podID="6a516a0a-7285-451c-be60-9f16f850c43c" containerID="fd578ca624a089b7bffeca07adff74b8ed6a4c7a4a2338436a73857a5d7f49b7" exitCode=0
Oct 30 19:54:27.329359 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.328774    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" event={"ID":"6a516a0a-7285-451c-be60-9f16f850c43c","Type":"ContainerDied","Data":"fd578ca624a089b7bffeca07adff74b8ed6a4c7a4a2338436a73857a5d7f49b7"}
Oct 30 19:54:27.334861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.331672    5085 generic.go:356] "Generic (PLEG): container finished" podID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerID="f5c1e3493e24e44b89df656a963a0b3217eea0075119dfc767da5a1468e81c9b" exitCode=0
Oct 30 19:54:27.334861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.331843    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerDied","Data":"f5c1e3493e24e44b89df656a963a0b3217eea0075119dfc767da5a1468e81c9b"}
Oct 30 19:54:27.334861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.331897    5085 scope.go:122] "RemoveContainer" containerID="64bb35533ab7d80e7f22fca5cadcf3d62b8e9edf685b3c44b7c20a1acad85ad6"
Oct 30 19:54:27.363222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.362860    5085 generic.go:356] "Generic (PLEG): container finished" podID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerID="0aec34dc0f8a294d62d713822a5363d50d8a0b498ec63bbb821f886d8c80c498" exitCode=0
Oct 30 19:54:27.363222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.363064    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" event={"ID":"89a50c7a-2ede-47ad-a22c-56796e8d3294","Type":"ContainerDied","Data":"0aec34dc0f8a294d62d713822a5363d50d8a0b498ec63bbb821f886d8c80c498"}
Oct 30 19:54:27.372738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.372590    5085 generic.go:356] "Generic (PLEG): container finished" podID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerID="53c9c163ab0a56a4ca7cc62616d1a554284682d0a36838dd7c9009d632e86bf3" exitCode=0
Oct 30 19:54:27.372738 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.372665    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" event={"ID":"ebaa0b35-1e80-44e0-a3cc-b77492e53a37","Type":"ContainerDied","Data":"53c9c163ab0a56a4ca7cc62616d1a554284682d0a36838dd7c9009d632e86bf3"}
Oct 30 19:54:27.450016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.449970    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:54:27.535755 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.535726    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.548734    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.548853    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.548925    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.548973    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.549010    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.549087    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.549275    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.549435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.549317    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.550192 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.550167    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\") pod \"6a516a0a-7285-451c-be60-9f16f850c43c\" (UID: \"6a516a0a-7285-451c-be60-9f16f850c43c\") "
Oct 30 19:54:27.551335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.551313    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.553252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.553228    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:27.553680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.553660    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.554075 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.554055    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.554465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.554447    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.568653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.568620    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.568844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.568824    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:27.573008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.572972    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.573521 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.573469    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm" pod "6a516a0a-7285-451c-be60-9f16f850c43c" (UID: "6a516a0a-7285-451c-be60-9f16f850c43c"). InnerVolumeSpecName "kube-api-access-slxjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.586725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.586689    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:54:27.589881 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.589853    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:54:27.651813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.651725    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.651813 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.651775    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652043 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.651892    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652043 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.651983    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652040    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652060    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.652148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652082    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652151    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652161    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.652304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652259    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.652304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652287    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652347    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\") pod \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\" (UID: \"ebaa0b35-1e80-44e0-a3cc-b77492e53a37\") "
Oct 30 19:54:27.652529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652377    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.652756 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.652733    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:27.653246 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653217    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653248    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a516a0a-7285-451c-be60-9f16f850c43c-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653265    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653280    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653294    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxjm\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-kube-api-access-slxjm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653311    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a516a0a-7285-451c-be60-9f16f850c43c-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653326    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653339    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653353    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a516a0a-7285-451c-be60-9f16f850c43c-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653371 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653367    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653382    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653396    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653409    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a516a0a-7285-451c-be60-9f16f850c43c-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.653927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.653423    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.654622 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.654596    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.654761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.654735    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "kube-api-access-rkd2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.654906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.654878    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:27.655064 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.655047    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token" pod "ebaa0b35-1e80-44e0-a3cc-b77492e53a37" (UID: "ebaa0b35-1e80-44e0-a3cc-b77492e53a37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.754036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754003    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754093    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754118    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754163    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754204    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754235    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754309    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754362    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754402    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754433    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.754455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754449    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754496    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754530    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754617    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754654    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754632    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754677    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.754828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754711    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754882    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754904    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\") pod \"89a50c7a-2ede-47ad-a22c-56796e8d3294\" (UID: \"89a50c7a-2ede-47ad-a22c-56796e8d3294\") "
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754951    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754999    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\") pod \"47e93fa0-525e-47a6-97a9-82089a0a7334\" (UID: \"47e93fa0-525e-47a6-97a9-82089a0a7334\") "
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.754993    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:27.755206 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.755114    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.755506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.755266    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.755506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.755370    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.755506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.755447    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.756020 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.755999    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756237 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756218    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756358 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756344    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756462 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756451    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756542    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756654    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756757    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e93fa0-525e-47a6-97a9-82089a0a7334-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756854    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.756965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.756952    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.757048 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.757036    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.757137 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.757123    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkd2c\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-kube-api-access-rkd2c\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.757220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.757208    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.757308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.757295    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebaa0b35-1e80-44e0-a3cc-b77492e53a37-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.757932 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.757908    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:27.759523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.759496    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "kube-api-access-kf9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.759941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.759765    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.759941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.759782    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.759941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.759839    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "kube-api-access-h697f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.759941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.759872    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:27.760438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.760415    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:27.761081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.761057    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo" pod "89a50c7a-2ede-47ad-a22c-56796e8d3294" (UID: "89a50c7a-2ede-47ad-a22c-56796e8d3294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:27.761215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.761196    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo" pod "47e93fa0-525e-47a6-97a9-82089a0a7334" (UID: "47e93fa0-525e-47a6-97a9-82089a0a7334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868727    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868770    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868795    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868811    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89a50c7a-2ede-47ad-a22c-56796e8d3294-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868827    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868842    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89a50c7a-2ede-47ad-a22c-56796e8d3294-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868863    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9pv\" (UniqueName: \"kubernetes.io/projected/47e93fa0-525e-47a6-97a9-82089a0a7334-kube-api-access-kf9pv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868881    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e93fa0-525e-47a6-97a9-82089a0a7334-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:27.871251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:27.868896    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h697f\" (UniqueName: \"kubernetes.io/projected/89a50c7a-2ede-47ad-a22c-56796e8d3294-kube-api-access-h697f\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:28.382129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.382046    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"
Oct 30 19:54:28.382129 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.382068    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th" event={"ID":"47e93fa0-525e-47a6-97a9-82089a0a7334","Type":"ContainerDied","Data":"065a3707fedc30f3cdd32cdc953c3be7d6fe0cca310a4108e6fec87a268be7b1"}
Oct 30 19:54:28.382698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.382155    5085 scope.go:122] "RemoveContainer" containerID="f5c1e3493e24e44b89df656a963a0b3217eea0075119dfc767da5a1468e81c9b"
Oct 30 19:54:28.383811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.383791    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"
Oct 30 19:54:28.383911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.383825    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn" event={"ID":"89a50c7a-2ede-47ad-a22c-56796e8d3294","Type":"ContainerDied","Data":"5ba5529db50bf8d20244b37714733b1b89081e508382a362ee061cf1e3200297"}
Oct 30 19:54:28.385574 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.385538    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"
Oct 30 19:54:28.385665 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.385558    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl" event={"ID":"ebaa0b35-1e80-44e0-a3cc-b77492e53a37","Type":"ContainerDied","Data":"c8263eb49d3d20eb923b99596a942ae991f33e42d4dc8f54cece42e5c6e179b7"}
Oct 30 19:54:28.387301 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.387269    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2" event={"ID":"6a516a0a-7285-451c-be60-9f16f850c43c","Type":"ContainerDied","Data":"25b92534c90ffac2353ed534fae7f97ed4e8bddb85c7a7cf8074546c3ee0ec9e"}
Oct 30 19:54:28.387394 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.387286    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"
Oct 30 19:54:28.397171 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.396851    5085 scope.go:122] "RemoveContainer" containerID="0aec34dc0f8a294d62d713822a5363d50d8a0b498ec63bbb821f886d8c80c498"
Oct 30 19:54:28.407927 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.407908    5085 scope.go:122] "RemoveContainer" containerID="53c9c163ab0a56a4ca7cc62616d1a554284682d0a36838dd7c9009d632e86bf3"
Oct 30 19:54:28.417168 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.417146    5085 scope.go:122] "RemoveContainer" containerID="fd578ca624a089b7bffeca07adff74b8ed6a4c7a4a2338436a73857a5d7f49b7"
Oct 30 19:54:28.543296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.543266    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:54:28.550067 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.550040    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-9bzhl"]
Oct 30 19:54:28.633070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.632991    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" path="/var/lib/kubelet/pods/ebaa0b35-1e80-44e0-a3cc-b77492e53a37/volumes"
Oct 30 19:54:28.657155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.657122    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:54:28.661993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.661970    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-rvnp2"]
Oct 30 19:54:28.766412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.766380    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:54:28.772033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.772003    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-sq8th"]
Oct 30 19:54:28.876489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.876457    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:54:28.882878 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:28.882850    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-khwpn"]
Oct 30 19:54:30.632431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:30.632394    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" path="/var/lib/kubelet/pods/47e93fa0-525e-47a6-97a9-82089a0a7334/volumes"
Oct 30 19:54:30.632933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:30.632916    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" path="/var/lib/kubelet/pods/6a516a0a-7285-451c-be60-9f16f850c43c/volumes"
Oct 30 19:54:30.633356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:30.633343    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" path="/var/lib/kubelet/pods/89a50c7a-2ede-47ad-a22c-56796e8d3294/volumes"
Oct 30 19:54:38.063350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.063312    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"]
Oct 30 19:54:38.063979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.063962    5085 cpu_manager.go:414] "containerMap: removing container" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy"
Oct 30 19:54:38.063979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.063979    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064007    5085 cpu_manager.go:414] "containerMap: removing container" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064015    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064037    5085 cpu_manager.go:414] "containerMap: removing container" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064042    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064054    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064059    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy"
Oct 30 19:54:38.064098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064099    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064106    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064286    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebaa0b35-1e80-44e0-a3cc-b77492e53a37" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064304    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a516a0a-7285-451c-be60-9f16f850c43c" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064311    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a50c7a-2ede-47ad-a22c-56796e8d3294" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064324    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.064347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.064336    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e93fa0-525e-47a6-97a9-82089a0a7334" containerName="istio-proxy"
Oct 30 19:54:38.067896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.067879    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.114656 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.114628    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrd6\" (UniqueName: \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.114800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.114744    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.114852 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.114822    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.133415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.133389    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"]
Oct 30 19:54:38.175173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.175112    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:54:38.175173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.175176    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:54:38.215812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.215785    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrd6\" (UniqueName: \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.215978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.215865    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.215978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.215953    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.216648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.216624    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.218365 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.218348    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.275025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.274992    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrd6\" (UniqueName: \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\") pod \"maas-api-revoke-keys-e2e-api-a-cf283d-xps68\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.381662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.381551    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:38.765613 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:38.765584    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"]
Oct 30 19:54:38.766993 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:54:38.766966    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cf92f7_10ea_45c0_a1a7_ee1c76ccb089.slice/crio-65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4 WatchSource:0}: Error finding container 65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4: Status 404 returned error can't find the container with id 65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4
Oct 30 19:54:39.502425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:39.502384    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68" event={"ID":"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089","Type":"ContainerStarted","Data":"18e9b9b9635d5356bf912a0c50b17002797a13ef1cc0e5be0f09a9d4c7649a96"}
Oct 30 19:54:39.502841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:39.502446    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68" event={"ID":"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089","Type":"ContainerStarted","Data":"65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4"}
Oct 30 19:54:39.613797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:39.613717    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68" podStartSLOduration=1.6136984559999998 podStartE2EDuration="1.613698456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:39.613083207 +0000 UTC m=+1789.117486964" watchObservedRunningTime="2026-10-30 19:54:39.613698456 +0000 UTC m=+1789.118102203"
Oct 30 19:54:40.511859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:40.511821    5085 generic.go:356] "Generic (PLEG): container finished" podID="27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" containerID="18e9b9b9635d5356bf912a0c50b17002797a13ef1cc0e5be0f09a9d4c7649a96" exitCode=0
Oct 30 19:54:40.512277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:40.511903    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68" event={"ID":"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089","Type":"ContainerDied","Data":"18e9b9b9635d5356bf912a0c50b17002797a13ef1cc0e5be0f09a9d4c7649a96"}
Oct 30 19:54:41.685855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.685830    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:41.764953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.764924    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\" (UniqueName: \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\") pod \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") "
Oct 30 19:54:41.765121 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.765059    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\") pod \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") "
Oct 30 19:54:41.765198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.765181    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\") pod \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\" (UID: \"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089\") "
Oct 30 19:54:41.765509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.765476    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca" pod "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" (UID: "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:41.765924 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.765902    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:41.767220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.767194    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6" pod "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" (UID: "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089"). InnerVolumeSpecName "kube-api-access-zhrd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:41.767220 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.767199    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca" pod "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" (UID: "27cf92f7-10ea-45c0-a1a7-ee1c76ccb089"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:54:41.867439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.867391    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:41.867439 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:41.867435    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrd6\" (UniqueName: \"kubernetes.io/projected/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089-kube-api-access-zhrd6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:42.531228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:42.531194    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68" event={"ID":"27cf92f7-10ea-45c0-a1a7-ee1c76ccb089","Type":"ContainerDied","Data":"65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4"}
Oct 30 19:54:42.531228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:42.531227    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"
Oct 30 19:54:42.531510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:42.531244    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c70d656ae8e6c7671e0d734642f9c4e06ee44b4d6ca28d00febb63f2ece8b4"
Oct 30 19:54:50.576628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:50.576588    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:54:51.127249 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.127216    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:54:51.130052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.130029    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:54:51.492317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.492275    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:54:51.492842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.492801    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" containerID="cri-o://0218a0934c7f59a36533f969a6c3f39c47c0cbd0020e366cf694a05fd5f31aca" gracePeriod=30
Oct 30 19:54:51.502069 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.498983    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:54:51.502069 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.499267    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" containerID="cri-o://22a6ca1a129720fe63b8bee950bf0e4897534002d7da8f1a3cb54a2add476fd5" gracePeriod=30
Oct 30 19:54:51.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.510857    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:54:51.511586 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.511086    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy" containerID="cri-o://1d18810d7e9372c6366129103b52d8cba6302412e01533871ad92b433c8c2451" gracePeriod=30
Oct 30 19:54:51.523584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.523537    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:54:51.546380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.523825    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" containerID="cri-o://5e54c3647d07f9af180976f25e322233fd3a636cc6a6ab78d4b1c9f1986e0052" gracePeriod=30
Oct 30 19:54:51.815410 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.815317    5085 scope.go:122] "RemoveContainer" containerID="0603f569e43af41fd1eb0a5562bf84e3f18c6d0b98a50931cd43f89d20b5c112"
Oct 30 19:54:51.830326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.830308    5085 scope.go:122] "RemoveContainer" containerID="0fd4741b39471dbc17b577086b511dbfd0e84cd0188046b3a64170dc474e2632"
Oct 30 19:54:51.843284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.843264    5085 scope.go:122] "RemoveContainer" containerID="a20fa2d02969951c8f3e92568ebf22c0ae1983cd0288ca6e9e498e627ada9465"
Oct 30 19:54:51.851712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.851690    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 19:54:51.852423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.852401    5085 cpu_manager.go:414] "containerMap: removing container" podUID="27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" containerName="revoke-keys"
Oct 30 19:54:51.852423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.852420    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" containerName="revoke-keys"
Oct 30 19:54:51.852583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.852554    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" containerName="revoke-keys"
Oct 30 19:54:51.854073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.854055    5085 scope.go:122] "RemoveContainer" containerID="0c8a72d0560cc3e1dbca7a9c86b37c60fa974ec6a03a89221d1f8140c52cc377"
Oct 30 19:54:51.873593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.858543    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926413    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926486    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926551    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926604    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926674    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926766    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926806    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926862    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.927401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.926893    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmjq\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:51.928222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.928200    5085 scope.go:122] "RemoveContainer" containerID="3a66c08afbd5ce4d2b2b563bc6d24befaf2ed7b05fb68c531fc24eb7c74b47e9"
Oct 30 19:54:51.928781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.928724    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-585635-openshift-default-dockercfg-79wdz\""
Oct 30 19:54:51.934116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.933947    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 19:54:51.942709 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.942525    5085 scope.go:122] "RemoveContainer" containerID="ed368d7fe8cce8380531664e1ced3fcdd508736cf3506dfa53791cd92380ad20"
Oct 30 19:54:51.953522 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:51.953496    5085 scope.go:122] "RemoveContainer" containerID="a15285ae4c72e5827bb2063868e74aee9c95d7196c8d4cbb03112c9c4a7e6fb5"
Oct 30 19:54:52.033756 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:54:52.020819    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:54:52.041181 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.041143    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.041536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.041515    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.041715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.041694    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.041896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.041879    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.042019 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.042006    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmjq\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.042185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.042168    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.042318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.042304    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.042472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.042458    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.042615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.042602    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.043589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.043527    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.043990 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.043969    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.044734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.044713    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.045323 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.045144    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.054151 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.053905    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.079284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.078968    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.101588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.091066    5085 scope.go:122] "RemoveContainer" containerID="50ec058530413c2d9c5629923b1c484841f59aab0480d27f0bcb2ba0990ba1ac"
Oct 30 19:54:52.101588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.099827    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.116147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.116095    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.116706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.116639    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmjq\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\") pod \"e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") " pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:52.143866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.143807    5085 scope.go:122] "RemoveContainer" containerID="e0faeef05e9b1afcc2e06dbdbd352aa9e88375501c0e38f8bf3d6e415b3834d4"
Oct 30 19:54:52.154420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.154396    5085 scope.go:122] "RemoveContainer" containerID="56d4f9252567b57963ce1b0d84dac3ad2e387e3ec01eadb1e0eee7aee8b04e78"
Oct 30 19:54:52.203169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:52.203143    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:53.138593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.138545    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 19:54:53.139505 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:54:53.139481    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81051a2_66d1_4944_a137_211dced84b5f.slice/crio-022e6f6c40c0550bfc35138d3155ca1e38de47581f5a8b82b07a65f0c221ec1a WatchSource:0}: Error finding container 022e6f6c40c0550bfc35138d3155ca1e38de47581f5a8b82b07a65f0c221ec1a: Status 404 returned error can't find the container with id 022e6f6c40c0550bfc35138d3155ca1e38de47581f5a8b82b07a65f0c221ec1a
Oct 30 19:54:53.142495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.142459    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:54:53.142603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.142541    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:54:53.142651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.142621    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:54:53.503798 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.503765    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:53.503967 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.503817    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:53.850840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.850746    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerStarted","Data":"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"}
Oct 30 19:54:53.850840 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:53.850800    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerStarted","Data":"022e6f6c40c0550bfc35138d3155ca1e38de47581f5a8b82b07a65f0c221ec1a"}
Oct 30 19:54:54.017353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.017299    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podStartSLOduration=3.017286531 podStartE2EDuration="3.017286531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:54.014679297 +0000 UTC m=+1803.519083049" watchObservedRunningTime="2026-10-30 19:54:54.017286531 +0000 UTC m=+1803.521690278"
Oct 30 19:54:54.086357 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.086313    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:54:54.086577 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.086540    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy" containerID="cri-o://bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c" gracePeriod=30
Oct 30 19:54:54.091472 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.091432    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:54:54.091722 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.091696    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy" containerID="cri-o://6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b" gracePeriod=30
Oct 30 19:54:54.095677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.095654    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:54:54.095835 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.095818    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" containerID="cri-o://e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895" gracePeriod=30
Oct 30 19:54:54.213589 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.204973    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:54.250014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.249984    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:54.868587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.865904    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:54.882138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:54.882116    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:54:55.581587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.580806    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" podUID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" containerName="payload-processing" containerID="cri-o://6aa98a5ed33cf867973b6287613071143bce282e195921f9400f70874e043e7f" gracePeriod=25
Oct 30 19:54:55.684427 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.684395    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:55.684623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.684449    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:55.895481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.895442    5085 generic.go:356] "Generic (PLEG): container finished" podID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" containerID="6aa98a5ed33cf867973b6287613071143bce282e195921f9400f70874e043e7f" exitCode=0
Oct 30 19:54:55.896685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.896651    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" event={"ID":"373e9c93-69a8-48ac-b3f3-b92da7d437f0","Type":"ContainerDied","Data":"6aa98a5ed33cf867973b6287613071143bce282e195921f9400f70874e043e7f"}
Oct 30 19:54:55.896842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.896713    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms" event={"ID":"373e9c93-69a8-48ac-b3f3-b92da7d437f0","Type":"ContainerDied","Data":"cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64"}
Oct 30 19:54:55.896842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.896732    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cee15bcb4a6e93715c6101037128f8d5e9ab8875257a2096e53fe512c6ab3e64"
Oct 30 19:54:55.903962 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.903934    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:54:55.987233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.987200    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\") pod \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") "
Oct 30 19:54:55.987415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.987268    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\" (UniqueName: \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\") pod \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\" (UID: \"373e9c93-69a8-48ac-b3f3-b92da7d437f0\") "
Oct 30 19:54:55.987596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.987574    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume" pod "373e9c93-69a8-48ac-b3f3-b92da7d437f0" (UID: "373e9c93-69a8-48ac-b3f3-b92da7d437f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:55.987983 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.987958    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373e9c93-69a8-48ac-b3f3-b92da7d437f0-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:55.989464 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:55.989441    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2" pod "373e9c93-69a8-48ac-b3f3-b92da7d437f0" (UID: "373e9c93-69a8-48ac-b3f3-b92da7d437f0"). InnerVolumeSpecName "kube-api-access-srff2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:56.090016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.089972    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srff2\" (UniqueName: \"kubernetes.io/projected/373e9c93-69a8-48ac-b3f3-b92da7d437f0-kube-api-access-srff2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:56.501623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.501589    5085 patch_prober.go:29] interesting pod/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:56.501790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.501681    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:56.920825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.920775    5085 generic.go:356] "Generic (PLEG): container finished" podID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerID="0218a0934c7f59a36533f969a6c3f39c47c0cbd0020e366cf694a05fd5f31aca" exitCode=0
Oct 30 19:54:56.921306 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.920957    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" event={"ID":"e49411fe-188a-4a92-896e-bb2dcdc1e302","Type":"ContainerDied","Data":"0218a0934c7f59a36533f969a6c3f39c47c0cbd0020e366cf694a05fd5f31aca"}
Oct 30 19:54:56.929555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.929525    5085 generic.go:356] "Generic (PLEG): container finished" podID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerID="1d18810d7e9372c6366129103b52d8cba6302412e01533871ad92b433c8c2451" exitCode=0
Oct 30 19:54:56.929746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.929621    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" event={"ID":"ebe38533-bc3c-4180-b45e-a10b909aef47","Type":"ContainerDied","Data":"1d18810d7e9372c6366129103b52d8cba6302412e01533871ad92b433c8c2451"}
Oct 30 19:54:56.933593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.932162    5085 generic.go:356] "Generic (PLEG): container finished" podID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerID="22a6ca1a129720fe63b8bee950bf0e4897534002d7da8f1a3cb54a2add476fd5" exitCode=0
Oct 30 19:54:56.933593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.932213    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" event={"ID":"dd90aeaf-df2d-4f71-8250-2989bebbe1bc","Type":"ContainerDied","Data":"22a6ca1a129720fe63b8bee950bf0e4897534002d7da8f1a3cb54a2add476fd5"}
Oct 30 19:54:56.934952 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.934931    5085 generic.go:356] "Generic (PLEG): container finished" podID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerID="5e54c3647d07f9af180976f25e322233fd3a636cc6a6ab78d4b1c9f1986e0052" exitCode=0
Oct 30 19:54:56.935103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.935058    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" event={"ID":"8604b620-8f84-40f0-8efc-229f8a352ec7","Type":"ContainerDied","Data":"5e54c3647d07f9af180976f25e322233fd3a636cc6a6ab78d4b1c9f1986e0052"}
Oct 30 19:54:56.935432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:56.935338    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"
Oct 30 19:54:57.013392 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.013266    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:54:57.062850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.062825    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:54:57.071824 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.071797    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:54:57.077393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.077369    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:54:57.120583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120520    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.120768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120682    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.120768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120716    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.120768 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120758    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120792    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120823    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120858    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120875    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120903    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.120976    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121037    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121072    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121093    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121128    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121168    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121193    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121220    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121252    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121285    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.121320 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121312    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121351    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121387    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121426    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121480    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121508    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121544    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121597    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121667    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\") pod \"e49411fe-188a-4a92-896e-bb2dcdc1e302\" (UID: \"e49411fe-188a-4a92-896e-bb2dcdc1e302\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121719    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121713    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121749    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121765    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.121815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121796    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121830    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121867    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121901    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\") pod \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\" (UID: \"dd90aeaf-df2d-4f71-8250-2989bebbe1bc\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121935    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.121963    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.122015    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.122042    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\") pod \"ebe38533-bc3c-4180-b45e-a10b909aef47\" (UID: \"ebe38533-bc3c-4180-b45e-a10b909aef47\") "
Oct 30 19:54:57.122259 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.122071    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\") pod \"8604b620-8f84-40f0-8efc-229f8a352ec7\" (UID: \"8604b620-8f84-40f0-8efc-229f8a352ec7\") "
Oct 30 19:54:57.122706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.122416    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.123061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123038    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123064    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123077    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123092    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123105    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123117    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe38533-bc3c-4180-b45e-a10b909aef47-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.123412 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123147    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:54:57.123543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123515    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.123725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123699    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:57.124044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:57.124044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.123955    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.124199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.124149    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.124404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.124382    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.124404 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.124392    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:57.125189 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.124715    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.126233 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.126022    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.127555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.127511    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.127555 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.127689    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:57.127939 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.127822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "kube-api-access-v4qm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.128265 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.128096    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.128772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.128380    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.128772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.128508    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "kube-api-access-2bzcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.129408 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.129181    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.130169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.130143    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cf283d-884ff6cc8-5bmms"]
Oct 30 19:54:57.130458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.130410    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.130980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.130928    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.131373 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.131346    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:57.132060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.132010    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.132332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.132292    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:57.132332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.132322    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.132485 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.132463    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "kube-api-access-rpdwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.132925 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.132897    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:57.133734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.133710    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy" pod "ebe38533-bc3c-4180-b45e-a10b909aef47" (UID: "ebe38533-bc3c-4180-b45e-a10b909aef47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.133859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.133835    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "kube-api-access-78dkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.133972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.133954    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token" pod "dd90aeaf-df2d-4f71-8250-2989bebbe1bc" (UID: "dd90aeaf-df2d-4f71-8250-2989bebbe1bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.135169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.135147    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token" pod "e49411fe-188a-4a92-896e-bb2dcdc1e302" (UID: "e49411fe-188a-4a92-896e-bb2dcdc1e302"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.135387 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.135356    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:57.135604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.135582    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy" pod "8604b620-8f84-40f0-8efc-229f8a352ec7" (UID: "8604b620-8f84-40f0-8efc-229f8a352ec7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:57.224338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224298    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdwd\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-kube-api-access-rpdwd\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224333    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e49411fe-188a-4a92-896e-bb2dcdc1e302-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224343    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224352    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224361    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224369    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bzcp\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-kube-api-access-2bzcp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224378    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224386    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224395    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224404    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224413    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224421    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224431    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224439    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8604b620-8f84-40f0-8efc-229f8a352ec7-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224448    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224458    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224465    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e49411fe-188a-4a92-896e-bb2dcdc1e302-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224473    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224481    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e49411fe-188a-4a92-896e-bb2dcdc1e302-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224489    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224499    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dkr\" (UniqueName: \"kubernetes.io/projected/8604b620-8f84-40f0-8efc-229f8a352ec7-kube-api-access-78dkr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224507    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224518    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224534    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224554    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd90aeaf-df2d-4f71-8250-2989bebbe1bc-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224589    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224598    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qm7\" (UniqueName: \"kubernetes.io/projected/ebe38533-bc3c-4180-b45e-a10b909aef47-kube-api-access-v4qm7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224607    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe38533-bc3c-4180-b45e-a10b909aef47-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224615    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe38533-bc3c-4180-b45e-a10b909aef47-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.224615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.224624    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8604b620-8f84-40f0-8efc-229f8a352ec7-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:57.954815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.954770    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd" event={"ID":"e49411fe-188a-4a92-896e-bb2dcdc1e302","Type":"ContainerDied","Data":"468315be26af4553c770a4a4cf8f5bc274e1866aa83dced253f918aa8a2ef171"}
Oct 30 19:54:57.955295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.954819    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"
Oct 30 19:54:57.955295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.954880    5085 scope.go:122] "RemoveContainer" containerID="0218a0934c7f59a36533f969a6c3f39c47c0cbd0020e366cf694a05fd5f31aca"
Oct 30 19:54:57.958604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.956699    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"
Oct 30 19:54:57.958604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.956727    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j" event={"ID":"ebe38533-bc3c-4180-b45e-a10b909aef47","Type":"ContainerDied","Data":"0272737e6baaaea6b7de777e4b5eed0ad579af1db4ee8d1479d3f09dd10e2ed4"}
Oct 30 19:54:57.962187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.962157    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"
Oct 30 19:54:57.962315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.962180    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87" event={"ID":"dd90aeaf-df2d-4f71-8250-2989bebbe1bc","Type":"ContainerDied","Data":"ad454d5deef43182ac96e6d1527c5a508af76666de89367903837b45a825bb38"}
Oct 30 19:54:57.963842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.963814    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md" event={"ID":"8604b620-8f84-40f0-8efc-229f8a352ec7","Type":"ContainerDied","Data":"6a7d46a0117a90218964afe1d0c3bfab34a68fbd12942ac97ba836b0174ceae7"}
Oct 30 19:54:57.963967 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.963863    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"
Oct 30 19:54:57.977219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.977200    5085 scope.go:122] "RemoveContainer" containerID="1d18810d7e9372c6366129103b52d8cba6302412e01533871ad92b433c8c2451"
Oct 30 19:54:57.990839 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:57.990812    5085 scope.go:122] "RemoveContainer" containerID="22a6ca1a129720fe63b8bee950bf0e4897534002d7da8f1a3cb54a2add476fd5"
Oct 30 19:54:58.001180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.001156    5085 scope.go:122] "RemoveContainer" containerID="5e54c3647d07f9af180976f25e322233fd3a636cc6a6ab78d4b1c9f1986e0052"
Oct 30 19:54:58.116750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.116704    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:54:58.120873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.120843    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-6mn87"]
Oct 30 19:54:58.227131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.227051    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:54:58.233465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.233438    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-7nrhd"]
Oct 30 19:54:58.340892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.340856    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:54:58.345992 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.345962    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-c422j"]
Oct 30 19:54:58.450801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.450766    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:54:58.456046 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.456009    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b0a183-openshift-default-657d79bfdc-qq7md"]
Oct 30 19:54:58.520481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.520400    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:54:58.520481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.520456    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:54:58.633849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.633813    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" path="/var/lib/kubelet/pods/373e9c93-69a8-48ac-b3f3-b92da7d437f0/volumes"
Oct 30 19:54:58.634452 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.634426    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" path="/var/lib/kubelet/pods/8604b620-8f84-40f0-8efc-229f8a352ec7/volumes"
Oct 30 19:54:58.635004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.634984    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" path="/var/lib/kubelet/pods/dd90aeaf-df2d-4f71-8250-2989bebbe1bc/volumes"
Oct 30 19:54:58.635849 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.635825    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" path="/var/lib/kubelet/pods/e49411fe-188a-4a92-896e-bb2dcdc1e302/volumes"
Oct 30 19:54:58.636340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.636321    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" path="/var/lib/kubelet/pods/ebe38533-bc3c-4180-b45e-a10b909aef47/volumes"
Oct 30 19:54:58.835786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.835696    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:58.835786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:58.835765    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:59.417587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.417247    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:54:59.468017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.467989    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:54:59.472775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.472749    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:54:59.558659 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.558623    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.558862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.558722    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.558862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.558780    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.558862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.558832    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.559222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559195    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.559314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559245    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.559314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559281    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.559434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559415    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.559495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559435    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.559495 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559469    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.559617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559518    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.559617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559547    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.559617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559580    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.559617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559590    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559626    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559659    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559721    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559750    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559789    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.559868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559847    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559899    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559928    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.559976    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560011    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560039    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560066    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.560179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560117    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.561038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560185    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\") pod \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\" (UID: \"73b38ca1-bbdc-4cb3-b849-486bf3cca50b\") "
Oct 30 19:54:59.561038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560221    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\") pod \"c81db512-33f1-40ea-aab2-0197a773fa4e\" (UID: \"c81db512-33f1-40ea-aab2-0197a773fa4e\") "
Oct 30 19:54:59.561038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.560298    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\") pod \"a3ad8d66-5114-4efc-a464-0132276abba3\" (UID: \"a3ad8d66-5114-4efc-a464-0132276abba3\") "
Oct 30 19:54:59.561343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.561243    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.561343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.561292    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.561343 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.561309    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.562096 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.561898    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.562356 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.562327    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:59.565303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.565006    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:59.565923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.565898    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.565990    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566215    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566287    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566299    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:59.566374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566337    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:59.566546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566490    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566508    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566647    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.566675 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566663    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "kube-api-access-znxp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.566975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566860    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.566975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566889    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.567588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.566983    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.567588 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.567024    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "kube-api-access-5q7rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.567790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.567756    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token" pod "c81db512-33f1-40ea-aab2-0197a773fa4e" (UID: "c81db512-33f1-40ea-aab2-0197a773fa4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.567960 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.567931    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.568203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.568172    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:59.568856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.568679    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.568856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.568772    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:59.568856 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.568823    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo" pod "a3ad8d66-5114-4efc-a464-0132276abba3" (UID: "a3ad8d66-5114-4efc-a464-0132276abba3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:59.570861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.570831    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4" pod "73b38ca1-bbdc-4cb3-b849-486bf3cca50b" (UID: "73b38ca1-bbdc-4cb3-b849-486bf3cca50b"). InnerVolumeSpecName "kube-api-access-vfxl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663155    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ad8d66-5114-4efc-a464-0132276abba3-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663193    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c81db512-33f1-40ea-aab2-0197a773fa4e-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663210    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663225    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663238    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663252    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663266    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663282    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663296    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663310    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663323    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663338    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663351    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663364    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7rl\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-kube-api-access-5q7rl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663379    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663392    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663405    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfxl4\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-kube-api-access-vfxl4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663418    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c81db512-33f1-40ea-aab2-0197a773fa4e-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663431    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxp2\" (UniqueName: \"kubernetes.io/projected/c81db512-33f1-40ea-aab2-0197a773fa4e-kube-api-access-znxp2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663445    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ad8d66-5114-4efc-a464-0132276abba3-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663458    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663470    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ad8d66-5114-4efc-a464-0132276abba3-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663484    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b38ca1-bbdc-4cb3-b849-486bf3cca50b-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.663587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.663497    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ad8d66-5114-4efc-a464-0132276abba3-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:54:59.987136 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.987105    5085 generic.go:356] "Generic (PLEG): container finished" podID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerID="bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c" exitCode=0
Oct 30 19:54:59.987309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.987174    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"
Oct 30 19:54:59.987309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.987195    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" event={"ID":"c81db512-33f1-40ea-aab2-0197a773fa4e","Type":"ContainerDied","Data":"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"}
Oct 30 19:54:59.987309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.987252    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66" event={"ID":"c81db512-33f1-40ea-aab2-0197a773fa4e","Type":"ContainerDied","Data":"d73a4b4257492178db8c33ce2b33177630a96c42b4470f58e66703a5debc6579"}
Oct 30 19:54:59.987309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.987275    5085 scope.go:122] "RemoveContainer" containerID="bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"
Oct 30 19:54:59.988896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.988876    5085 generic.go:356] "Generic (PLEG): container finished" podID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerID="e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895" exitCode=0
Oct 30 19:54:59.989000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.988943    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"
Oct 30 19:54:59.989000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.988951    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" event={"ID":"73b38ca1-bbdc-4cb3-b849-486bf3cca50b","Type":"ContainerDied","Data":"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"}
Oct 30 19:54:59.989000 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.988988    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb" event={"ID":"73b38ca1-bbdc-4cb3-b849-486bf3cca50b","Type":"ContainerDied","Data":"dfddd22cf45ce5a01243738950ec2db37fb35a28a0230c3988d612a8c3b04952"}
Oct 30 19:54:59.990691 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.990669    5085 generic.go:356] "Generic (PLEG): container finished" podID="a3ad8d66-5114-4efc-a464-0132276abba3" containerID="6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b" exitCode=0
Oct 30 19:54:59.990801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.990719    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"
Oct 30 19:54:59.990801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.990730    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" event={"ID":"a3ad8d66-5114-4efc-a464-0132276abba3","Type":"ContainerDied","Data":"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"}
Oct 30 19:54:59.990801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:54:59.990771    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h" event={"ID":"a3ad8d66-5114-4efc-a464-0132276abba3","Type":"ContainerDied","Data":"1c6fdf9bcb6caba95e7c285f0e75ecb3eba0239dc64767e6e559d1f9bbf3c164"}
Oct 30 19:55:00.020119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.019974    5085 scope.go:122] "RemoveContainer" containerID="bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"
Oct 30 19:55:00.027402 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:00.027364    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c\": container with ID starting with bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c not found: ID does not exist" containerID="bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"
Oct 30 19:55:00.027614 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.027409    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c"} err="failed to get container status \"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c\": rpc error: code = NotFound desc = could not find container \"bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c\": container with ID starting with bec297995caec45fe6e3ed0bcecc5aa59cf5508742dadcb76eff3a16bdc71c1c not found: ID does not exist"
Oct 30 19:55:00.027614 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.027432    5085 scope.go:122] "RemoveContainer" containerID="e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"
Oct 30 19:55:00.046305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.046280    5085 scope.go:122] "RemoveContainer" containerID="e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"
Oct 30 19:55:00.047710 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:00.047677    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895\": container with ID starting with e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895 not found: ID does not exist" containerID="e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"
Oct 30 19:55:00.047950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.047717    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895"} err="failed to get container status \"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895\": rpc error: code = NotFound desc = could not find container \"e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895\": container with ID starting with e248dff1d490f2398ba95efc92ba9c8a1c68df940ac22c5bb445af5ac4e1b895 not found: ID does not exist"
Oct 30 19:55:00.047950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.047736    5085 scope.go:122] "RemoveContainer" containerID="6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"
Oct 30 19:55:00.066662 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.066642    5085 scope.go:122] "RemoveContainer" containerID="6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"
Oct 30 19:55:00.067048 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:00.067030    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b\": container with ID starting with 6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b not found: ID does not exist" containerID="6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"
Oct 30 19:55:00.067120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.067056    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b"} err="failed to get container status \"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b\": rpc error: code = NotFound desc = could not find container \"6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b\": container with ID starting with 6c757e9d6181b4acff9a3c27d76e4d2ff93e8e7b509637f35c990417a4aaa26b not found: ID does not exist"
Oct 30 19:55:00.157155 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.157118    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:55:00.168284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.168252    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-zfs66"]
Oct 30 19:55:00.274496 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.274367    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:55:00.294073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.294040    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-wh4vb"]
Oct 30 19:55:00.384640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.383407    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:55:00.400634 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.394720    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-7vc7h"]
Oct 30 19:55:00.653511 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.653416    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" path="/var/lib/kubelet/pods/73b38ca1-bbdc-4cb3-b849-486bf3cca50b/volumes"
Oct 30 19:55:00.654239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.654215    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" path="/var/lib/kubelet/pods/a3ad8d66-5114-4efc-a464-0132276abba3/volumes"
Oct 30 19:55:00.654906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.654881    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" path="/var/lib/kubelet/pods/c81db512-33f1-40ea-aab2-0197a773fa4e/volumes"
Oct 30 19:55:00.843295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.842992    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843840    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843861    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843885    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843893    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843916    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843922    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843936    5085 cpu_manager.go:414] "containerMap: removing container" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843943    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843961    5085 cpu_manager.go:414] "containerMap: removing container" podUID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" containerName="payload-processing"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843967    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" containerName="payload-processing"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843978    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843984    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.843995    5085 cpu_manager.go:414] "containerMap: removing container" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844001    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844023    5085 cpu_manager.go:414] "containerMap: removing container" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844029    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844171    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ad8d66-5114-4efc-a464-0132276abba3" containerName="istio-proxy"
Oct 30 19:55:00.844165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844184    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c81db512-33f1-40ea-aab2-0197a773fa4e" containerName="istio-proxy"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844199    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe38533-bc3c-4180-b45e-a10b909aef47" containerName="istio-proxy"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844209    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd90aeaf-df2d-4f71-8250-2989bebbe1bc" containerName="istio-proxy"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844218    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="373e9c93-69a8-48ac-b3f3-b92da7d437f0" containerName="payload-processing"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844225    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49411fe-188a-4a92-896e-bb2dcdc1e302" containerName="istio-proxy"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844233    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b38ca1-bbdc-4cb3-b849-486bf3cca50b" containerName="istio-proxy"
Oct 30 19:55:00.845134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.844245    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="8604b620-8f84-40f0-8efc-229f8a352ec7" containerName="istio-proxy"
Oct 30 19:55:00.852580 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.852531    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:00.887217 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.887181    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:55:00.903835 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.903762    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-585635\""
Oct 30 19:55:00.904005 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.903913    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-585635-dockercfg-cb6tp\""
Oct 30 19:55:00.985418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.985383    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:55:00.989825 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:00.989802    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.031168 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.031112    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64kg\" (UniqueName: \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.031326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.031306    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.046620 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.046594    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:55:01.132772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.132732    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25ls\" (UniqueName: \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.134138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.132906    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.134138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.132951    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.134138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.133077    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64kg\" (UniqueName: \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.134138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.133584    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.192396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.192369    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64kg\" (UniqueName: \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\") pod \"payload-processing-e2e-api-b-585635-5598c7d4d-f4982\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") " pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.234871 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.234840    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.235037 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.235020    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25ls\" (UniqueName: \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.235625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.235607    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.293831 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.293805    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25ls\" (UniqueName: \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\") pod \"payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.314399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.314372    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:01.470467 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.470384    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:01.693554 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.693528    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:55:01.695156 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:55:01.695127    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90dd14bc_ef0f_49e4_ad7a_2ac957aaed72.slice/crio-3cd407f9831642899df5167c5fbb8ecd5968f93dc2dcd4a784552e8c25310204 WatchSource:0}: Error finding container 3cd407f9831642899df5167c5fbb8ecd5968f93dc2dcd4a784552e8c25310204: Status 404 returned error can't find the container with id 3cd407f9831642899df5167c5fbb8ecd5968f93dc2dcd4a784552e8c25310204
Oct 30 19:55:01.844894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:01.844865    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:55:01.845678 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:55:01.845645    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0ee7a3_eb42_4c72_913e_1efccd4fe559.slice/crio-1d58d5b1053643a3451a48002efd0a212bcce155fecee979fb2c9744a9e4a208 WatchSource:0}: Error finding container 1d58d5b1053643a3451a48002efd0a212bcce155fecee979fb2c9744a9e4a208: Status 404 returned error can't find the container with id 1d58d5b1053643a3451a48002efd0a212bcce155fecee979fb2c9744a9e4a208
Oct 30 19:55:02.017188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.017089    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" event={"ID":"ca0ee7a3-eb42-4c72-913e-1efccd4fe559","Type":"ContainerStarted","Data":"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"}
Oct 30 19:55:02.017188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.017151    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" event={"ID":"ca0ee7a3-eb42-4c72-913e-1efccd4fe559","Type":"ContainerStarted","Data":"1d58d5b1053643a3451a48002efd0a212bcce155fecee979fb2c9744a9e4a208"}
Oct 30 19:55:02.017372 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.017200    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:02.019303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.019281    5085 generic.go:356] "Generic (PLEG): container finished" podID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerID="0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736" exitCode=0
Oct 30 19:55:02.019430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.019353    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerDied","Data":"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"}
Oct 30 19:55:02.019828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.019801    5085 scope.go:122] "RemoveContainer" containerID="0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"
Oct 30 19:55:02.021414 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021389    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" event={"ID":"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72","Type":"ContainerStarted","Data":"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"}
Oct 30 19:55:02.021516 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021438    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" event={"ID":"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72","Type":"ContainerStarted","Data":"3cd407f9831642899df5167c5fbb8ecd5968f93dc2dcd4a784552e8c25310204"}
Oct 30 19:55:02.021617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021558    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:02.021729 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021702    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:55:02.021794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021774    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:55:02.021848 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.021813    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:55:02.131830 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.131778    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" podStartSLOduration=2.131763258 podStartE2EDuration="2.131763258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:02.128253967 +0000 UTC m=+1811.632657714" watchObservedRunningTime="2026-10-30 19:55:02.131763258 +0000 UTC m=+1811.636167004"
Oct 30 19:55:02.372246 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:02.372144    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" podStartSLOduration=2.372126803 podStartE2EDuration="2.372126803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:02.370655411 +0000 UTC m=+1811.875059163" watchObservedRunningTime="2026-10-30 19:55:02.372126803 +0000 UTC m=+1811.876530550"
Oct 30 19:55:03.032007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.031967    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerStarted","Data":"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"}
Oct 30 19:55:03.289231 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.289149    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-1062c2"]
Oct 30 19:55:03.293640 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.293611    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:03.344148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.344119    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\""
Oct 30 19:55:03.344544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.344524    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-1062c2"]
Oct 30 19:55:03.481636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.480989    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:55:03.481636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.481151    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:55:03.481636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.481193    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:55:03.487313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.487286    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9snp\" (UniqueName: \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\") pod \"test-curl-7337-1062c2\" (UID: \"7ef39240-5725-4ff3-80da-470661511e25\") " pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:03.588483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.588402    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9snp\" (UniqueName: \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\") pod \"test-curl-7337-1062c2\" (UID: \"7ef39240-5725-4ff3-80da-470661511e25\") " pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:03.647497 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.647457    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9snp\" (UniqueName: \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\") pod \"test-curl-7337-1062c2\" (UID: \"7ef39240-5725-4ff3-80da-470661511e25\") " pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:03.906850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:03.906773    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:04.300325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:04.300296    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-1062c2"]
Oct 30 19:55:04.465973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:04.465931    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:55:04.466238 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:04.466211    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:55:05.056068 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:05.056027    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-1062c2" event={"ID":"7ef39240-5725-4ff3-80da-470661511e25","Type":"ContainerStarted","Data":"1ded63587ae22cf3fd7d23dab777bfae1b8e1ee98571c7534f3b89112b8bb57a"}
Oct 30 19:55:05.479587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:05.478988    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:55:05.479587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:05.479050    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:55:06.482590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:06.478627    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:55:06.482590 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:06.478686    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:55:07.104052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:07.104006    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-1062c2" event={"ID":"7ef39240-5725-4ff3-80da-470661511e25","Type":"ContainerStarted","Data":"64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248"}
Oct 30 19:55:07.214785 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:07.214722    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7337-1062c2" podStartSLOduration=2.017213382 podStartE2EDuration="4.214707395s" totalImagesPullingTime="2.197494013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:07.213077194 +0000 UTC m=+1816.717480941" watchObservedRunningTime="2026-10-30 19:55:07.214707395 +0000 UTC m=+1816.719111142"
Oct 30 19:55:07.468022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:07.467983    5085 patch_prober.go:29] interesting pod/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 19:55:07.468292 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:07.468261    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 19:55:08.470256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:08.470227    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:55:08.470715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:08.470665    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:55:08.471434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:08.471415    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:55:08.729215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:08.729130    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-1062c2"]
Oct 30 19:55:08.735805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:08.735772    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-1062c2"]
Oct 30 19:55:09.118339 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.118244    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7337-1062c2" podUID="7ef39240-5725-4ff3-80da-470661511e25" containerName="test-curl-7337-1062c2" containerID="cri-o://64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248" gracePeriod=2
Oct 30 19:55:09.167078 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:09.167041    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-1062c2\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="7ef39240-5725-4ff3-80da-470661511e25" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:09.743745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.743714    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"]
Oct 30 19:55:09.744370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.744318    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7ef39240-5725-4ff3-80da-470661511e25" containerName="test-curl-7337-1062c2"
Oct 30 19:55:09.744370 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.744333    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef39240-5725-4ff3-80da-470661511e25" containerName="test-curl-7337-1062c2"
Oct 30 19:55:09.744536 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.744521    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef39240-5725-4ff3-80da-470661511e25" containerName="test-curl-7337-1062c2"
Oct 30 19:55:09.748278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.748261    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.798817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.798772    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:09.798996 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.798953    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:55:09.803202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.803145    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"]
Oct 30 19:55:09.860822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.860789    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljlr\" (UniqueName: \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.860998 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.860963    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.861085 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.861070    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.906571 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:09.906518    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-1062c2\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="7ef39240-5725-4ff3-80da-470661511e25" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:09.962473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.962436    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.962655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.962537    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljlr\" (UniqueName: \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.962710 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.962656    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.963309 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.963291    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:09.964822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:09.964803    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:10.021761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:10.021683    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljlr\" (UniqueName: \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-99cn2\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:10.061771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:10.061745    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:10.439252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:10.439221    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"]
Oct 30 19:55:10.439406 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:55:10.439300    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2925bca0_51ce_4256_91b4_e205ba1f9a8b.slice/crio-334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242 WatchSource:0}: Error finding container 334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242: Status 404 returned error can't find the container with id 334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242
Oct 30 19:55:10.676902 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:10.676870    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-1062c2\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="7ef39240-5725-4ff3-80da-470661511e25" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:11.140629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.140600    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2" event={"ID":"2925bca0-51ce-4256-91b4-e205ba1f9a8b","Type":"ContainerStarted","Data":"2187b718306551621a872538a36d2567d8b32a9eda2ad989231baeef618be402"}
Oct 30 19:55:11.140935 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.140679    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2" event={"ID":"2925bca0-51ce-4256-91b4-e205ba1f9a8b","Type":"ContainerStarted","Data":"334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242"}
Oct 30 19:55:11.255295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.255245    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2" podStartSLOduration=2.255229488 podStartE2EDuration="2.255229488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.251834906 +0000 UTC m=+1820.756238653" watchObservedRunningTime="2026-10-30 19:55:11.255229488 +0000 UTC m=+1820.759633231"
Oct 30 19:55:11.370383 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.370354    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:11.481109 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.481081    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\" (UniqueName: \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\") pod \"7ef39240-5725-4ff3-80da-470661511e25\" (UID: \"7ef39240-5725-4ff3-80da-470661511e25\") "
Oct 30 19:55:11.483088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.483063    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp" pod "7ef39240-5725-4ff3-80da-470661511e25" (UID: "7ef39240-5725-4ff3-80da-470661511e25"). InnerVolumeSpecName "kube-api-access-p9snp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:11.582454 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:11.582421    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9snp\" (UniqueName: \"kubernetes.io/projected/7ef39240-5725-4ff3-80da-470661511e25-kube-api-access-p9snp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:12.149038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.149005    5085 generic.go:356] "Generic (PLEG): container finished" podID="7ef39240-5725-4ff3-80da-470661511e25" containerID="64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248" exitCode=137
Oct 30 19:55:12.149477 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.149049    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:12.149477 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.149091    5085 scope.go:122] "RemoveContainer" containerID="64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248"
Oct 30 19:55:12.150994 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.150969    5085 generic.go:356] "Generic (PLEG): container finished" podID="2925bca0-51ce-4256-91b4-e205ba1f9a8b" containerID="2187b718306551621a872538a36d2567d8b32a9eda2ad989231baeef618be402" exitCode=0
Oct 30 19:55:12.151106 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.151021    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2" event={"ID":"2925bca0-51ce-4256-91b4-e205ba1f9a8b","Type":"ContainerDied","Data":"2187b718306551621a872538a36d2567d8b32a9eda2ad989231baeef618be402"}
Oct 30 19:55:12.159473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.159455    5085 scope.go:122] "RemoveContainer" containerID="64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248"
Oct 30 19:55:12.159746 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:12.159729    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248\": container with ID starting with 64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248 not found: ID does not exist" containerID="64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248"
Oct 30 19:55:12.159807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.159753    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248"} err="failed to get container status \"64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248\": rpc error: code = NotFound desc = could not find container \"64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248\": container with ID starting with 64a91fd318f80e017df770d9b04244d91d0730832722e20d41f42cf463712248 not found: ID does not exist"
Oct 30 19:55:12.312010 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:12.311923    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-1062c2\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="7ef39240-5725-4ff3-80da-470661511e25" pod="opendatahub/test-curl-7337-1062c2"
Oct 30 19:55:12.632242 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:12.632165    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ef39240-5725-4ff3-80da-470661511e25" path="/var/lib/kubelet/pods/7ef39240-5725-4ff3-80da-470661511e25/volumes"
Oct 30 19:55:13.035665 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.035636    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:55:13.035816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.035702    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:55:13.353841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.353805    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:13.403775 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.403740    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\") pod \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") "
Oct 30 19:55:13.403936 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.403868    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\") pod \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") "
Oct 30 19:55:13.404118 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.404099    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\" (UniqueName: \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\") pod \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\" (UID: \"2925bca0-51ce-4256-91b4-e205ba1f9a8b\") "
Oct 30 19:55:13.404163 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.404119    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca" pod "2925bca0-51ce-4256-91b4-e205ba1f9a8b" (UID: "2925bca0-51ce-4256-91b4-e205ba1f9a8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:13.404530 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.404509    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:13.405989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.405965    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca" pod "2925bca0-51ce-4256-91b4-e205ba1f9a8b" (UID: "2925bca0-51ce-4256-91b4-e205ba1f9a8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:13.406124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.406098    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr" pod "2925bca0-51ce-4256-91b4-e205ba1f9a8b" (UID: "2925bca0-51ce-4256-91b4-e205ba1f9a8b"). InnerVolumeSpecName "kube-api-access-xljlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:13.505816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.505771    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljlr\" (UniqueName: \"kubernetes.io/projected/2925bca0-51ce-4256-91b4-e205ba1f9a8b-kube-api-access-xljlr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:13.505816 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:13.505816    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2925bca0-51ce-4256-91b4-e205ba1f9a8b-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:14.168617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:14.168592    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"
Oct 30 19:55:14.168787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:14.168622    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2" event={"ID":"2925bca0-51ce-4256-91b4-e205ba1f9a8b","Type":"ContainerDied","Data":"334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242"}
Oct 30 19:55:14.168787 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:14.168675    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="334610b8c9ea9aab278bd0826569e30d60eb58ccb44422dbafcd8692553bd242"
Oct 30 19:55:17.510205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:17.510169    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"]
Oct 30 19:55:17.557381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:17.557346    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8d3b6f-p9fn7"]
Oct 30 19:55:18.631795 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:18.631761    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ccc9475-115c-4fe1-b3e1-706a8d47dbd3" path="/var/lib/kubelet/pods/7ccc9475-115c-4fe1-b3e1-706a8d47dbd3/volumes"
Oct 30 19:55:20.328599 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:20.328534    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:55:21.940087 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:21.940049    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:55:24.934198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:24.934166    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:55:24.934793 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:24.934374    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy" containerID="cri-o://f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c" gracePeriod=30
Oct 30 19:55:25.804848 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.804811    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"]
Oct 30 19:55:25.805424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.805411    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2925bca0-51ce-4256-91b4-e205ba1f9a8b" containerName="revoke-keys"
Oct 30 19:55:25.805469 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.805425    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2925bca0-51ce-4256-91b4-e205ba1f9a8b" containerName="revoke-keys"
Oct 30 19:55:25.805532 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.805523    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2925bca0-51ce-4256-91b4-e205ba1f9a8b" containerName="revoke-keys"
Oct 30 19:55:25.810442 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.810421    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:25.864225 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.864192    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"]
Oct 30 19:55:25.906605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.906574    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:55:25.906605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.906596    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:25.999869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.999835    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.000275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:25.999949    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.000275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.000045    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhjh\" (UniqueName: \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.101061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.100978    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.101061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.101048    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.101253 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.101200    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhjh\" (UniqueName: \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.101841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.101816    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.103271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.103244    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.160465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.160430    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhjh\" (UniqueName: \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\") pod \"maas-api-revoke-keys-e2e-api-b-4477e2-tflgn\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.424703 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.424674    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:26.803558 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:26.803529    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"]
Oct 30 19:55:26.804246 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:55:26.804213    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f7dfd4_843d_4e53_8e1e_cde74d8ae38b.slice/crio-6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7 WatchSource:0}: Error finding container 6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7: Status 404 returned error can't find the container with id 6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7
Oct 30 19:55:27.272043 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:27.272001    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn" event={"ID":"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b","Type":"ContainerStarted","Data":"8b5326df5f95296ef078fc55fa4358837de9844bd19649d135e8cf1c5b093271"}
Oct 30 19:55:27.272438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:27.272052    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn" event={"ID":"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b","Type":"ContainerStarted","Data":"6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7"}
Oct 30 19:55:27.387314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:27.387215    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn" podStartSLOduration=2.387200519 podStartE2EDuration="2.387200519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:27.383918651 +0000 UTC m=+1836.888322425" watchObservedRunningTime="2026-10-30 19:55:27.387200519 +0000 UTC m=+1836.891604266"
Oct 30 19:55:28.281702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:28.281669    5085 generic.go:356] "Generic (PLEG): container finished" podID="f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" containerID="8b5326df5f95296ef078fc55fa4358837de9844bd19649d135e8cf1c5b093271" exitCode=0
Oct 30 19:55:28.282083 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:28.281758    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn" event={"ID":"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b","Type":"ContainerDied","Data":"8b5326df5f95296ef078fc55fa4358837de9844bd19649d135e8cf1c5b093271"}
Oct 30 19:55:29.469296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.469268    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:29.546223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.546146    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\") pod \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") "
Oct 30 19:55:29.546376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.546244    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\") pod \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") "
Oct 30 19:55:29.546376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.546284    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\" (UniqueName: \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\") pod \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\" (UID: \"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b\") "
Oct 30 19:55:29.546608 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.546581    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca" pod "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" (UID: "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:29.548289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.548267    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca" pod "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" (UID: "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:29.548419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.548400    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh" pod "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" (UID: "f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b"). InnerVolumeSpecName "kube-api-access-bqhjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:29.648432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.648397    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:29.648432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.648421    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:29.648432 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:29.648434    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhjh\" (UniqueName: \"kubernetes.io/projected/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b-kube-api-access-bqhjh\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.201892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.201832    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:55:30.255277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255249    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255328    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255424    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255500    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255554    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255674    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255747    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255805 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255775    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:30.255911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255808    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:30.255911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255817    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255874    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\") pod \"b0668d04-5116-4911-9668-763e3b6e0e55\" (UID: \"b0668d04-5116-4911-9668-763e3b6e0e55\") "
Oct 30 19:55:30.255911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255896    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:30.256102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.255909    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:30.256102 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256005    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:30.256451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256402    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.256451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256424    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.256451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256439    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.256451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256452    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0668d04-5116-4911-9668-763e3b6e0e55-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.256661 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.256465    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.257706 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.257685    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:30.257788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.257708    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:30.258281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.258261    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "kube-api-access-md2vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:30.258333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.258287    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token" pod "b0668d04-5116-4911-9668-763e3b6e0e55" (UID: "b0668d04-5116-4911-9668-763e3b6e0e55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:30.298176 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.298154    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"
Oct 30 19:55:30.298303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.298149    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn" event={"ID":"f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b","Type":"ContainerDied","Data":"6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7"}
Oct 30 19:55:30.298303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.298280    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6310e2e659c1b8a77ba929c3bd47197b274c2964fbc14680b4dadd14e9f747d7"
Oct 30 19:55:30.300164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.300140    5085 generic.go:356] "Generic (PLEG): container finished" podID="b0668d04-5116-4911-9668-763e3b6e0e55" containerID="f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c" exitCode=0
Oct 30 19:55:30.300273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.300179    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"
Oct 30 19:55:30.300273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.300218    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerDied","Data":"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"}
Oct 30 19:55:30.300273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.300264    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt" event={"ID":"b0668d04-5116-4911-9668-763e3b6e0e55","Type":"ContainerDied","Data":"8c8311e414729c9255f9ba6659cd58e00e9d6e68a3d15e21e10437131d5a8040"}
Oct 30 19:55:30.300379 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.300289    5085 scope.go:122] "RemoveContainer" containerID="f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"
Oct 30 19:55:30.312618 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.312599    5085 scope.go:122] "RemoveContainer" containerID="7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"
Oct 30 19:55:30.321709 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.321694    5085 scope.go:122] "RemoveContainer" containerID="f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"
Oct 30 19:55:30.322012 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:30.321990    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c\": container with ID starting with f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c not found: ID does not exist" containerID="f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"
Oct 30 19:55:30.322078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.322021    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c"} err="failed to get container status \"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c\": rpc error: code = NotFound desc = could not find container \"f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c\": container with ID starting with f476302dad0205e5464531860d3ded03d6c87f6d75280faae8b20acb9792d74c not found: ID does not exist"
Oct 30 19:55:30.322078 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.322040    5085 scope.go:122] "RemoveContainer" containerID="7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"
Oct 30 19:55:30.322277 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:30.322262    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c\": container with ID starting with 7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c not found: ID does not exist" containerID="7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"
Oct 30 19:55:30.322317 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.322282    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c"} err="failed to get container status \"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c\": rpc error: code = NotFound desc = could not find container \"7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c\": container with ID starting with 7318a5add5ac316d5f43ebe13d70b020359a526aeb9d3f5f4d6da05f3ff9105c not found: ID does not exist"
Oct 30 19:55:30.357322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.357295    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2vc\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-kube-api-access-md2vc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.357322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.357320    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0668d04-5116-4911-9668-763e3b6e0e55-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.357473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.357330    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0668d04-5116-4911-9668-763e3b6e0e55-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.357473 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.357339    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0668d04-5116-4911-9668-763e3b6e0e55-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.474678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.474605    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:55:30.481815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.481787    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8d3b6f-openshift-default-75785f64b7-c5mrt"]
Oct 30 19:55:30.633527 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:30.633479    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" path="/var/lib/kubelet/pods/b0668d04-5116-4911-9668-763e3b6e0e55/volumes"
Oct 30 19:55:45.049769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:45.049736    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"]
Oct 30 19:55:45.096861 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:45.096829    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa160ed-c6cbp"]
Oct 30 19:55:46.048712 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:46.048681    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"]
Oct 30 19:55:46.096060 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:46.096025    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30bb01c0-k46gn"]
Oct 30 19:55:46.640061 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:46.640027    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e7eea7-7bcc-4066-9518-894367d4190f" path="/var/lib/kubelet/pods/27e7eea7-7bcc-4066-9518-894367d4190f/volumes"
Oct 30 19:55:46.640477 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:46.640463    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee" path="/var/lib/kubelet/pods/50247cbc-9e0b-45f0-aa15-6b5e4c0b1cee/volumes"
Oct 30 19:55:49.560131 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.560098    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"]
Oct 30 19:55:49.560977 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.560954    5085 cpu_manager.go:414] "containerMap: removing container" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.560977 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.560976    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.561229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.560995    5085 cpu_manager.go:414] "containerMap: removing container" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.561229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561003    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.561229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561078    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" containerName="revoke-keys"
Oct 30 19:55:49.561229 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561086    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" containerName="revoke-keys"
Oct 30 19:55:49.561421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561271    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" containerName="revoke-keys"
Oct 30 19:55:49.561421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561285    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.561421 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.561296    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0668d04-5116-4911-9668-763e3b6e0e55" containerName="istio-proxy"
Oct 30 19:55:49.566244 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.566224    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.616669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.616644    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:55:49.620919 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.620889    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"]
Oct 30 19:55:49.662140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.662110    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:55:49.713108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.713079    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.713325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.713304    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87df2\" (UniqueName: \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.713450 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.713423    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.815093 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.815007    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.815272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.815142    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87df2\" (UniqueName: \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.815272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.815198    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.815809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.815786    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.817610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.817586    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.874437 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.874404    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87df2\" (UniqueName: \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\") pod \"maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:49.880247 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:49.880229    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:50.257420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:50.257395    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"]
Oct 30 19:55:50.258954 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:55:50.258916    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b07d55_688f_4e2c_976e_81917a916367.slice/crio-c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46 WatchSource:0}: Error finding container c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46: Status 404 returned error can't find the container with id c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46
Oct 30 19:55:50.470807 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:50.470770    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4" event={"ID":"97b07d55-688f-4e2c-976e-81917a916367","Type":"ContainerStarted","Data":"c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46"}
Oct 30 19:55:51.479537 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:51.479504    5085 generic.go:356] "Generic (PLEG): container finished" podID="97b07d55-688f-4e2c-976e-81917a916367" containerID="94073c8733aa680c2f20081b548e754193d71175909c19f51aaf5527bd8574f9" exitCode=0
Oct 30 19:55:51.479973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:51.479604    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4" event={"ID":"97b07d55-688f-4e2c-976e-81917a916367","Type":"ContainerDied","Data":"94073c8733aa680c2f20081b548e754193d71175909c19f51aaf5527bd8574f9"}
Oct 30 19:55:51.939220 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:55:51.939187    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:55:52.649173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.649144    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:52.751004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.750933    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\") pod \"97b07d55-688f-4e2c-976e-81917a916367\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") "
Oct 30 19:55:52.751004 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.750984    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\" (UniqueName: \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\") pod \"97b07d55-688f-4e2c-976e-81917a916367\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") "
Oct 30 19:55:52.751240 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.751021    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\") pod \"97b07d55-688f-4e2c-976e-81917a916367\" (UID: \"97b07d55-688f-4e2c-976e-81917a916367\") "
Oct 30 19:55:52.751303 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.751258    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca" pod "97b07d55-688f-4e2c-976e-81917a916367" (UID: "97b07d55-688f-4e2c-976e-81917a916367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:52.751551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.751533    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97b07d55-688f-4e2c-976e-81917a916367-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:52.753107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.753083    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca" pod "97b07d55-688f-4e2c-976e-81917a916367" (UID: "97b07d55-688f-4e2c-976e-81917a916367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:52.753205 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.753161    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2" pod "97b07d55-688f-4e2c-976e-81917a916367" (UID: "97b07d55-688f-4e2c-976e-81917a916367"). InnerVolumeSpecName "kube-api-access-87df2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:52.850550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.850517    5085 scope.go:122] "RemoveContainer" containerID="1ff0a578b2ca3fa341351d0485dde681a12590256ea888ea3e913a57cd924ecd"
Oct 30 19:55:52.852363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.852338    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87df2\" (UniqueName: \"kubernetes.io/projected/97b07d55-688f-4e2c-976e-81917a916367-kube-api-access-87df2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:52.852363 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.852366    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97b07d55-688f-4e2c-976e-81917a916367-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:55:52.861084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:52.861062    5085 scope.go:122] "RemoveContainer" containerID="dc198700eba6e8013fb825b4856d593f6034e01eadaaf4f5fbb1b48c596c3795"
Oct 30 19:55:53.095669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:53.095598    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:55:53.496888 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:53.496859    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"
Oct 30 19:55:53.497051 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:53.496901    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4" event={"ID":"97b07d55-688f-4e2c-976e-81917a916367","Type":"ContainerDied","Data":"c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46"}
Oct 30 19:55:53.497051 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:55:53.496957    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62f2231d0d9133bc265fa02986d20cc892376478a87c4becb8f50dcfe0fcc46"
Oct 30 19:56:07.036876 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.036835    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 19:56:07.037538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.037419    5085 cpu_manager.go:414] "containerMap: removing container" podUID="97b07d55-688f-4e2c-976e-81917a916367" containerName="revoke-keys"
Oct 30 19:56:07.037538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.037432    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b07d55-688f-4e2c-976e-81917a916367" containerName="revoke-keys"
Oct 30 19:56:07.037732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.037649    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b07d55-688f-4e2c-976e-81917a916367" containerName="revoke-keys"
Oct 30 19:56:07.043544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.043519    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.096728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.096693    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 19:56:07.140081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.140046    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-585635-serving-cert\""
Oct 30 19:56:07.140081 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.140072    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:56:07.140327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.140167    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-skrr4\""
Oct 30 19:56:07.155961 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.155930    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.156113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.156086    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9wb\" (UniqueName: \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.156157 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.156136    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.156210 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.156198    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.257751 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.257703    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.257945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.257821    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9wb\" (UniqueName: \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.257945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.257841    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.257945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.257894    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.258082 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:56:07.257977    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-585635-serving-cert: secret "maas-api-e2e-api-b-585635-serving-cert" not found
Oct 30 19:56:07.258082 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:56:07.258054    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls podName:d3295acc-a9f4-4c59-be56-426ac5c7cbcf nodeName:}" failed. No retries permitted until 2026-10-30 19:56:07.75803254 +0000 UTC m=+1877.262436265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls") pod "maas-api-e2e-api-b-585635-68859d554c-vwgld" (UID: "d3295acc-a9f4-4c59-be56-426ac5c7cbcf") : secret "maas-api-e2e-api-b-585635-serving-cert" not found
Oct 30 19:56:07.258654 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.258613    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.260218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.260199    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.317641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.317524    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9wb\" (UniqueName: \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.778239 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.778194    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.780651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.780623    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") pod \"maas-api-e2e-api-b-585635-68859d554c-vwgld\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:07.958777 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:07.958735    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:08.337602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:08.337575    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 19:56:08.338238 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:56:08.338207    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3295acc_a9f4_4c59_be56_426ac5c7cbcf.slice/crio-10bb8416b53ea476d228ccb03b16a145bb042d6bb7005443db90523af7a672b0 WatchSource:0}: Error finding container 10bb8416b53ea476d228ccb03b16a145bb042d6bb7005443db90523af7a672b0: Status 404 returned error can't find the container with id 10bb8416b53ea476d228ccb03b16a145bb042d6bb7005443db90523af7a672b0
Oct 30 19:56:08.609794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:08.609713    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" event={"ID":"d3295acc-a9f4-4c59-be56-426ac5c7cbcf","Type":"ContainerStarted","Data":"10bb8416b53ea476d228ccb03b16a145bb042d6bb7005443db90523af7a672b0"}
Oct 30 19:56:09.048693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:09.048663    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"]
Oct 30 19:56:09.095820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:09.095788    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a66c7306-fgq47"]
Oct 30 19:56:10.634685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:10.634650    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0" path="/var/lib/kubelet/pods/ce61b513-c882-4c3e-8b84-5f2cbd3c3ac0/volumes"
Oct 30 19:56:11.643591 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:11.643531    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" event={"ID":"d3295acc-a9f4-4c59-be56-426ac5c7cbcf","Type":"ContainerStarted","Data":"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"}
Oct 30 19:56:11.643953 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:11.643600    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:11.756542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:11.756491    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" podStartSLOduration=3.148932776 podStartE2EDuration="5.756477176s" totalImagesPullingTime="2.6075444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:11.755938906 +0000 UTC m=+1881.260342653" watchObservedRunningTime="2026-10-30 19:56:11.756477176 +0000 UTC m=+1881.260880923"
Oct 30 19:56:17.048156 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.048115    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"]
Oct 30 19:56:17.052725 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.052700    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.107275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.107244    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"]
Oct 30 19:56:17.150025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.149991    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:17.150025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.150007    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:56:17.202845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.202814    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.203018 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.202855    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhjv\" (UniqueName: \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.203147 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.203123    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.304596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.304482    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.304596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.304541    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.304596 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.304590    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhjv\" (UniqueName: \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.305304 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.305280    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.306954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.306935    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.363752 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.363719    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhjv\" (UniqueName: \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\") pod \"maas-api-revoke-sub-01027a28-jb77h\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.366538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.366511    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:17.660041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.659955    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 19:56:17.748295 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:17.748249    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"]
Oct 30 19:56:18.698028 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:18.697988    5085 generic.go:356] "Generic (PLEG): container finished" podID="f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" containerID="41c271a323c34ebf34daadc400f98e9b9fa014765d0dd329570b27a99de12a7c" exitCode=0
Oct 30 19:56:18.698407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:18.698026    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h" event={"ID":"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f","Type":"ContainerDied","Data":"41c271a323c34ebf34daadc400f98e9b9fa014765d0dd329570b27a99de12a7c"}
Oct 30 19:56:18.698407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:18.698085    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h" event={"ID":"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f","Type":"ContainerStarted","Data":"c1cc99d8bc59a7544137580b031a4366609e0ae708155b5911b1d8b17a17b6b3"}
Oct 30 19:56:19.867786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:19.867758    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:20.042730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.042650    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\" (UniqueName: \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\") pod \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") "
Oct 30 19:56:20.042730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.042709    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\") pod \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") "
Oct 30 19:56:20.042974 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.042836    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\") pod \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\" (UID: \"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f\") "
Oct 30 19:56:20.043223 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.043199    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca" pod "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" (UID: "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:20.043360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.043344    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:20.044788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.044760    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca" pod "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" (UID: "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:20.044897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.044845    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv" pod "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" (UID: "f1c1a2d6-9f58-42f2-983d-1e28a8beef2f"). InnerVolumeSpecName "kube-api-access-4dhjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:20.144642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.144610    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dhjv\" (UniqueName: \"kubernetes.io/projected/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-kube-api-access-4dhjv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:20.144642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.144639    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:20.584185 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.584153    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:56:20.584420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.584392    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy" containerID="cri-o://4eeaf6609b8daf8b4c12bf15e3797ec31e4a0b0164a5e27cf7a3a4ad57467ac3" gracePeriod=30
Oct 30 19:56:20.714549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.714519    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"
Oct 30 19:56:20.714724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.714554    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h" event={"ID":"f1c1a2d6-9f58-42f2-983d-1e28a8beef2f","Type":"ContainerDied","Data":"c1cc99d8bc59a7544137580b031a4366609e0ae708155b5911b1d8b17a17b6b3"}
Oct 30 19:56:20.714724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:20.714615    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cc99d8bc59a7544137580b031a4366609e0ae708155b5911b1d8b17a17b6b3"
Oct 30 19:56:21.939986 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:56:21.939946    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:56:24.586897 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:24.586863    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"]
Oct 30 19:56:24.635008 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:24.634976    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-cf283d-xps68"]
Oct 30 19:56:25.755324 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.755285    5085 generic.go:356] "Generic (PLEG): container finished" podID="43905552-d8e5-43a8-a012-062a18155336" containerID="4eeaf6609b8daf8b4c12bf15e3797ec31e4a0b0164a5e27cf7a3a4ad57467ac3" exitCode=0
Oct 30 19:56:25.755704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.755352    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" event={"ID":"43905552-d8e5-43a8-a012-062a18155336","Type":"ContainerDied","Data":"4eeaf6609b8daf8b4c12bf15e3797ec31e4a0b0164a5e27cf7a3a4ad57467ac3"}
Oct 30 19:56:25.867103 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.867076    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:56:25.926017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.925985    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926023    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926051    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926150    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926188    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926275    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926303    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:25.926399 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926362    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926404    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926625 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926435    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\") pod \"43905552-d8e5-43a8-a012-062a18155336\" (UID: \"43905552-d8e5-43a8-a012-062a18155336\") "
Oct 30 19:56:25.926727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926616    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:25.926727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926707    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:25.926886 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926867    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:25.926965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.926942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:25.927138 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.927123    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43905552-d8e5-43a8-a012-062a18155336-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.927208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.927144    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.927208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.927157    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.927208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.927171    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.927208 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.927184    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:25.928465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.928435    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "kube-api-access-vp7dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:25.928617 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.928501    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:25.928746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.928730    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:25.928802 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:25.928770    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token" pod "43905552-d8e5-43a8-a012-062a18155336" (UID: "43905552-d8e5-43a8-a012-062a18155336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:26.028010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.027973    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43905552-d8e5-43a8-a012-062a18155336-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.028010 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.028006    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7dz\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-kube-api-access-vp7dz\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.028209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.028021    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43905552-d8e5-43a8-a012-062a18155336-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.028209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.028033    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43905552-d8e5-43a8-a012-062a18155336-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:26.632107 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.632076    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27cf92f7-10ea-45c0-a1a7-ee1c76ccb089" path="/var/lib/kubelet/pods/27cf92f7-10ea-45c0-a1a7-ee1c76ccb089/volumes"
Oct 30 19:56:26.772794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.772758    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"
Oct 30 19:56:26.772794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.772782    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp" event={"ID":"43905552-d8e5-43a8-a012-062a18155336","Type":"ContainerDied","Data":"54706f485866b43e0d5d3166e27e0a426a7c8955c2ec9a80cc188ce4eea33de0"}
Oct 30 19:56:26.773279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.772863    5085 scope.go:122] "RemoveContainer" containerID="4eeaf6609b8daf8b4c12bf15e3797ec31e4a0b0164a5e27cf7a3a4ad57467ac3"
Oct 30 19:56:26.980504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:26.980467    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:56:27.028190 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:27.028155    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-rz6cp"]
Oct 30 19:56:28.632430 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:28.632397    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43905552-d8e5-43a8-a012-062a18155336" path="/var/lib/kubelet/pods/43905552-d8e5-43a8-a012-062a18155336/volumes"
Oct 30 19:56:31.615355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:31.615321    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:56:31.615915 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:31.615628    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy" containerID="cri-o://0d7c2e19eaf207a3775e0a73a0bf2acbb5ab04bf3832634d0ba64b0bdd39bfcb" gracePeriod=30
Oct 30 19:56:32.104823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.104791    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:56:32.496698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.496667    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"]
Oct 30 19:56:32.498015 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.497988    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" containerName="revoke-keys"
Oct 30 19:56:32.498215 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.498195    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" containerName="revoke-keys"
Oct 30 19:56:32.506748 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.506714    5085 cpu_manager.go:414] "containerMap: removing container" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy"
Oct 30 19:56:32.506945 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.506928    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy"
Oct 30 19:56:32.507351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.507322    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="43905552-d8e5-43a8-a012-062a18155336" containerName="istio-proxy"
Oct 30 19:56:32.507500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.507488    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" containerName="revoke-keys"
Oct 30 19:56:32.534157 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.534116    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.545610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.545538    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"]
Oct 30 19:56:32.588160 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.588123    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:56:32.590779 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.590746    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:56:32.619318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.619288    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbzw2\" (UniqueName: \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.619705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.619471    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.619705 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.619501    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.720517 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.720471    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.720719 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.720529    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.720812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.720785    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbzw2\" (UniqueName: \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.721345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.721298    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.722968 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.722942    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.780283 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.780209    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbzw2\" (UniqueName: \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\") pod \"maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:32.895251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:32.895218    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:33.286315 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:56:33.286269    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cdba21_2590_456e_8d9d_400d60adc640.slice/crio-6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec WatchSource:0}: Error finding container 6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec: Status 404 returned error can't find the container with id 6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec
Oct 30 19:56:33.286448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:33.286365    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"]
Oct 30 19:56:33.835677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:33.835640    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj" event={"ID":"32cdba21-2590-456e-8d9d-400d60adc640","Type":"ContainerStarted","Data":"e760bac54dd350767a83c49ce84a769137016c51a5f76370f44d452e0f8378b9"}
Oct 30 19:56:33.836123 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:33.835698    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj" event={"ID":"32cdba21-2590-456e-8d9d-400d60adc640","Type":"ContainerStarted","Data":"6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec"}
Oct 30 19:56:33.947694 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:33.947596    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj" podStartSLOduration=1.947582113 podStartE2EDuration="1.947582113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:33.945246534 +0000 UTC m=+1903.449650281" watchObservedRunningTime="2026-10-30 19:56:33.947582113 +0000 UTC m=+1903.451985859"
Oct 30 19:56:34.844820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:34.844782    5085 generic.go:356] "Generic (PLEG): container finished" podID="32cdba21-2590-456e-8d9d-400d60adc640" containerID="e760bac54dd350767a83c49ce84a769137016c51a5f76370f44d452e0f8378b9" exitCode=0
Oct 30 19:56:34.845228 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:34.844868    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj" event={"ID":"32cdba21-2590-456e-8d9d-400d60adc640","Type":"ContainerDied","Data":"e760bac54dd350767a83c49ce84a769137016c51a5f76370f44d452e0f8378b9"}
Oct 30 19:56:36.032204 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.032180    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:36.187736 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.187703    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\") pod \"32cdba21-2590-456e-8d9d-400d60adc640\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") "
Oct 30 19:56:36.187926 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.187746    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\" (UniqueName: \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\") pod \"32cdba21-2590-456e-8d9d-400d60adc640\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") "
Oct 30 19:56:36.187994 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.187953    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\") pod \"32cdba21-2590-456e-8d9d-400d60adc640\" (UID: \"32cdba21-2590-456e-8d9d-400d60adc640\") "
Oct 30 19:56:36.188542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.188507    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca" pod "32cdba21-2590-456e-8d9d-400d60adc640" (UID: "32cdba21-2590-456e-8d9d-400d60adc640"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:36.190522 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.190492    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca" pod "32cdba21-2590-456e-8d9d-400d60adc640" (UID: "32cdba21-2590-456e-8d9d-400d60adc640"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:56:36.191011 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.190987    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2" pod "32cdba21-2590-456e-8d9d-400d60adc640" (UID: "32cdba21-2590-456e-8d9d-400d60adc640"). InnerVolumeSpecName "kube-api-access-rbzw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:36.289481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.289444    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.289481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.289473    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbzw2\" (UniqueName: \"kubernetes.io/projected/32cdba21-2590-456e-8d9d-400d60adc640-kube-api-access-rbzw2\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.289481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.289482    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32cdba21-2590-456e-8d9d-400d60adc640-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.892345 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.892266    5085 generic.go:356] "Generic (PLEG): container finished" podID="f7bb5ea7-61af-488a-b80b-89c213321588" containerID="0d7c2e19eaf207a3775e0a73a0bf2acbb5ab04bf3832634d0ba64b0bdd39bfcb" exitCode=0
Oct 30 19:56:36.892506 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.892467    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerDied","Data":"0d7c2e19eaf207a3775e0a73a0bf2acbb5ab04bf3832634d0ba64b0bdd39bfcb"}
Oct 30 19:56:36.892639 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.892527    5085 scope.go:122] "RemoveContainer" containerID="422032546a36ced680b596c5855a8a8559ee130ed6382da4e1c184831d7605aa"
Oct 30 19:56:36.898724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.898698    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj" event={"ID":"32cdba21-2590-456e-8d9d-400d60adc640","Type":"ContainerDied","Data":"6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec"}
Oct 30 19:56:36.898826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.898740    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be571bc48bbce44a65e389aa0412dee77dad95262978e16e2d17393708fefec"
Oct 30 19:56:36.898826 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.898794    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"
Oct 30 19:56:36.906419 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.906395    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:56:36.998728 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998700    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.998940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998778    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.998940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998804    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.998940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998836    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.998940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998878    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.999166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.998947    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.999166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999025    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.999166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999113    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.999325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999178    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\") pod \"f7bb5ea7-61af-488a-b80b-89c213321588\" (UID: \"f7bb5ea7-61af-488a-b80b-89c213321588\") "
Oct 30 19:56:36.999325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999189    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:36.999325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999210    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:36.999325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999255    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:36.999523 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999392    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:36.999593 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999533    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:36.999744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999717    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7bb5ea7-61af-488a-b80b-89c213321588-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.999810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999748    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.999810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999762    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.999810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999776    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:36.999810 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:36.999789    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.001156 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.001130    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:37.001378 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.001353    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:37.001508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.001490    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "kube-api-access-zdd8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:37.001630 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.001613    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token" pod "f7bb5ea7-61af-488a-b80b-89c213321588" (UID: "f7bb5ea7-61af-488a-b80b-89c213321588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:37.103666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.101264    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.103666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.101296    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdd8x\" (UniqueName: \"kubernetes.io/projected/f7bb5ea7-61af-488a-b80b-89c213321588-kube-api-access-zdd8x\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.103666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.101311    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7bb5ea7-61af-488a-b80b-89c213321588-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.103666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.101325    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7bb5ea7-61af-488a-b80b-89c213321588-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:37.908628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.908602    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"
Oct 30 19:56:37.909347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.908601    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq" event={"ID":"f7bb5ea7-61af-488a-b80b-89c213321588","Type":"ContainerDied","Data":"244444a90041246ef303b439fceec2802eb0a1ea897afb2af671cb5fd8ccbc64"}
Oct 30 19:56:37.909347 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:37.908731    5085 scope.go:122] "RemoveContainer" containerID="0d7c2e19eaf207a3775e0a73a0bf2acbb5ab04bf3832634d0ba64b0bdd39bfcb"
Oct 30 19:56:38.151482 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:38.151449    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:56:38.181583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:38.177799    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cf283d-openshift-default-7c6cfc484f-fflqq"]
Oct 30 19:56:38.632838 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:38.632767    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" path="/var/lib/kubelet/pods/f7bb5ea7-61af-488a-b80b-89c213321588/volumes"
Oct 30 19:56:40.906770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:40.906723    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:56:40.907179 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:40.906808    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:56:41.942175 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.942138    5085 generic.go:356] "Generic (PLEG): container finished" podID="a81051a2-66d1-4944-a137-211dced84b5f" containerID="fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48" exitCode=0
Oct 30 19:56:41.942636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.942218    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerDied","Data":"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"}
Oct 30 19:56:41.942695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.942679    5085 scope.go:122] "RemoveContainer" containerID="fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"
Oct 30 19:56:41.944203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.944174    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:56:41.944260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.944240    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:56:41.944305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:41.944271    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:56:42.984112 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:42.984074    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerStarted","Data":"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"}
Oct 30 19:56:43.203942 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:43.203907    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:56:43.204136 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:43.204113    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:56:43.204192 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:43.204163    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 19:56:44.203823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:44.203790    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:56:44.204203 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:44.203844    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 19:56:45.204629 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:45.204588    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:56:45.205014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:45.204657    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 19:56:46.204579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:46.204524    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:56:46.204762 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:46.204613    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 19:56:47.204116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:47.204080    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body=
Oct 30 19:56:47.204340 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:47.204135    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused"
Oct 30 19:56:48.304492 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:48.304449    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:56:48.305845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:48.305678    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:56:48.306335 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:48.306310    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 19:56:51.939744 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:56:51.939702    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:56:52.938584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:52.938535    5085 scope.go:122] "RemoveContainer" containerID="6aa98a5ed33cf867973b6287613071143bce282e195921f9400f70874e043e7f"
Oct 30 19:56:52.948938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:52.948915    5085 scope.go:122] "RemoveContainer" containerID="331bf0e57448de8594d7f287c8d2392055f25db517ffda63dc62416a38be1b13"
Oct 30 19:56:52.957809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:52.957794    5085 scope.go:122] "RemoveContainer" containerID="daed6e5c3015d457785c7f822c2509d41e7e417cf58f1be235e408c225d930c4"
Oct 30 19:56:54.487698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:54.487662    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:56:54.488134 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:54.487870    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" containerID="cri-o://919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd" gracePeriod=30
Oct 30 19:56:58.954595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:58.954501    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-tzs66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:56:58.955002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:58.954596    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:56:59.751715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.751691    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:56:59.929883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.929853    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.929904    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.929977    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930020    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930097    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930175    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930178    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:59.930261 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930252    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930363    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930481 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930408    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\") pod \"69148e2d-793c-4165-ab14-29013735444f\" (UID: \"69148e2d-793c-4165-ab14-29013735444f\") "
Oct 30 19:56:59.930626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930511    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:59.930626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930518    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:59.930747 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930729    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:59.930815 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.930766    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:59.931084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.931059    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:59.931218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.931087    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:59.931218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.931102    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:59.931218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.931116    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:59.931218 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.931131    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69148e2d-793c-4165-ab14-29013735444f-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:56:59.933142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.933112    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:56:59.933142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.933133    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:56:59.933316 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.933160    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "kube-api-access-4h427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:59.933316 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:56:59.933256    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token" pod "69148e2d-793c-4165-ab14-29013735444f" (UID: "69148e2d-793c-4165-ab14-29013735444f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:00.032281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.032242    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h427\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-kube-api-access-4h427\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.032281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.032276    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69148e2d-793c-4165-ab14-29013735444f-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.032281 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.032286    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69148e2d-793c-4165-ab14-29013735444f-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.032761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.032295    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69148e2d-793c-4165-ab14-29013735444f-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.111667 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.111625    5085 generic.go:356] "Generic (PLEG): container finished" podID="69148e2d-793c-4165-ab14-29013735444f" containerID="919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd" exitCode=0
Oct 30 19:57:00.111842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.111723    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"
Oct 30 19:57:00.111842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.111732    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" event={"ID":"69148e2d-793c-4165-ab14-29013735444f","Type":"ContainerDied","Data":"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"}
Oct 30 19:57:00.111842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.111796    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66" event={"ID":"69148e2d-793c-4165-ab14-29013735444f","Type":"ContainerDied","Data":"5bd3f394275c072fd23108f41e1aef0e837f72cbf24f2b18f33febc4af801505"}
Oct 30 19:57:00.111842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.111830    5085 scope.go:122] "RemoveContainer" containerID="919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"
Oct 30 19:57:00.123681 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.123662    5085 scope.go:122] "RemoveContainer" containerID="919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"
Oct 30 19:57:00.123950 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:00.123933    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd\": container with ID starting with 919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd not found: ID does not exist" containerID="919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"
Oct 30 19:57:00.124002 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.123959    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd"} err="failed to get container status \"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd\": rpc error: code = NotFound desc = could not find container \"919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd\": container with ID starting with 919e2e12f01d4ccf02a3339719933f5ff912ea087a5efd6f2467fe9bc8c9d8cd not found: ID does not exist"
Oct 30 19:57:00.140582 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:00.140549    5085 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69148e2d_793c_4165_ab14_29013735444f.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 19:57:00.270987 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.270907    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:57:00.275535 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.275509    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-tzs66"]
Oct 30 19:57:00.631828 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:00.631752    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69148e2d-793c-4165-ab14-29013735444f" path="/var/lib/kubelet/pods/69148e2d-793c-4165-ab14-29013735444f/volumes"
Oct 30 19:57:03.333832 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:03.333795    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:57:09.493202 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:09.493164    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:57:09.493844 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:09.493426    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" containerID="cri-o://b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1" gracePeriod=30
Oct 30 19:57:11.254333 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254301    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"]
Oct 30 19:57:11.254879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254864    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:11.254947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254880    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:11.254947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254918    5085 cpu_manager.go:414] "containerMap: removing container" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy"
Oct 30 19:57:11.254947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254923    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy"
Oct 30 19:57:11.254947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254939    5085 cpu_manager.go:414] "containerMap: removing container" podUID="32cdba21-2590-456e-8d9d-400d60adc640" containerName="revoke-keys"
Oct 30 19:57:11.254947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.254943    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cdba21-2590-456e-8d9d-400d60adc640" containerName="revoke-keys"
Oct 30 19:57:11.255146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.255048    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:11.255146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.255067    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:11.255146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.255080    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="69148e2d-793c-4165-ab14-29013735444f" containerName="istio-proxy"
Oct 30 19:57:11.255146 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.255088    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cdba21-2590-456e-8d9d-400d60adc640" containerName="revoke-keys"
Oct 30 19:57:11.258684 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.258665    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.312381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.312349    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"]
Oct 30 19:57:11.355334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.355294    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:11.355483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.355433    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:57:11.369649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.369623    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.369794 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.369657    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flr8m\" (UniqueName: \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.369842 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.369807    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.471505 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.471467    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.471745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.471621    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.471745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.471646    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flr8m\" (UniqueName: \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.472326 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.472302    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.473890 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.473865    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.537290 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.537200    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flr8m\" (UniqueName: \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\") pod \"maas-api-revoke-keys-e2e-api-a-69932e-qjbgp\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.572933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.572899    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:11.947866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:11.947837    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"]
Oct 30 19:57:11.949200 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:57:11.949169    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607ac483_bdbd_45f1_b159_04fb3e2b2267.slice/crio-358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04 WatchSource:0}: Error finding container 358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04: Status 404 returned error can't find the container with id 358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04
Oct 30 19:57:12.227188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:12.227113    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp" event={"ID":"607ac483-bdbd-45f1-b159-04fb3e2b2267","Type":"ContainerStarted","Data":"358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04"}
Oct 30 19:57:13.235628 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:13.235591    5085 generic.go:356] "Generic (PLEG): container finished" podID="607ac483-bdbd-45f1-b159-04fb3e2b2267" containerID="bbdea3f30c1b36793b6fccb14ed99a53f009f55a847afc45371fddf42945c9b0" exitCode=0
Oct 30 19:57:13.235991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:13.235677    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp" event={"ID":"607ac483-bdbd-45f1-b159-04fb3e2b2267","Type":"ContainerDied","Data":"bbdea3f30c1b36793b6fccb14ed99a53f009f55a847afc45371fddf42945c9b0"}
Oct 30 19:57:13.740655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:13.740615    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-4vfdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:57:13.740823 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:13.740676    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:57:14.407260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.407233    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:14.508866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.508801    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\") pod \"607ac483-bdbd-45f1-b159-04fb3e2b2267\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") "
Oct 30 19:57:14.508866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.508850    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\" (UniqueName: \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\") pod \"607ac483-bdbd-45f1-b159-04fb3e2b2267\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") "
Oct 30 19:57:14.509119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.509099    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\") pod \"607ac483-bdbd-45f1-b159-04fb3e2b2267\" (UID: \"607ac483-bdbd-45f1-b159-04fb3e2b2267\") "
Oct 30 19:57:14.509546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.509511    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca" pod "607ac483-bdbd-45f1-b159-04fb3e2b2267" (UID: "607ac483-bdbd-45f1-b159-04fb3e2b2267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:14.511780 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.511724    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m" pod "607ac483-bdbd-45f1-b159-04fb3e2b2267" (UID: "607ac483-bdbd-45f1-b159-04fb3e2b2267"). InnerVolumeSpecName "kube-api-access-flr8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:14.511883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.511823    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca" pod "607ac483-bdbd-45f1-b159-04fb3e2b2267" (UID: "607ac483-bdbd-45f1-b159-04fb3e2b2267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:14.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.611105    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.611134    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/607ac483-bdbd-45f1-b159-04fb3e2b2267-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.611143    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flr8m\" (UniqueName: \"kubernetes.io/projected/607ac483-bdbd-45f1-b159-04fb3e2b2267-kube-api-access-flr8m\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.754514 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.754489    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:57:14.813746 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813672    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.813909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813814    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.813909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813879    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813915    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813936    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.814032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.813976    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814048    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814126    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814157    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.814359 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814215    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814359 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814277    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\") pod \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\" (UID: \"82afc92c-3c63-42d8-88cc-39b82e11f0f2\") "
Oct 30 19:57:14.814359 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814304    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.814504 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814436    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:14.814636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814612    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.815017 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.814992    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.815120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.815022    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.815120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.815039    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.815120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.815053    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.815120 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.815067    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.816068 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.816041    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:14.816167 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.816145    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:14.816234 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.816214    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:14.816302 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.816287    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6" pod "82afc92c-3c63-42d8-88cc-39b82e11f0f2" (UID: "82afc92c-3c63-42d8-88cc-39b82e11f0f2"). InnerVolumeSpecName "kube-api-access-khvx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:14.920636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.920601    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.920636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.920633    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.920636 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.920642    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:14.920862 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:14.920651    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khvx6\" (UniqueName: \"kubernetes.io/projected/82afc92c-3c63-42d8-88cc-39b82e11f0f2-kube-api-access-khvx6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:15.252653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.252613    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp" event={"ID":"607ac483-bdbd-45f1-b159-04fb3e2b2267","Type":"ContainerDied","Data":"358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04"}
Oct 30 19:57:15.252653 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.252646    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"
Oct 30 19:57:15.252931 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.252700    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358449b02e0fe862d465aea970cb7e79f74d30083450bb51e0077e003187aa04"
Oct 30 19:57:15.254289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.254259    5085 generic.go:356] "Generic (PLEG): container finished" podID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerID="b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1" exitCode=0
Oct 30 19:57:15.254448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.254333    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"
Oct 30 19:57:15.254448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.254336    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" event={"ID":"82afc92c-3c63-42d8-88cc-39b82e11f0f2","Type":"ContainerDied","Data":"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"}
Oct 30 19:57:15.254448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.254388    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh" event={"ID":"82afc92c-3c63-42d8-88cc-39b82e11f0f2","Type":"ContainerDied","Data":"0c798e40adc149a405c2b74ff26fd1a27181f1bd98e58ef173973aa92f7567fa"}
Oct 30 19:57:15.254448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.254421    5085 scope.go:122] "RemoveContainer" containerID="b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"
Oct 30 19:57:15.265870 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.265847    5085 scope.go:122] "RemoveContainer" containerID="b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"
Oct 30 19:57:15.266143 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:15.266117    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1\": container with ID starting with b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1 not found: ID does not exist" containerID="b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"
Oct 30 19:57:15.266199 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.266156    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1"} err="failed to get container status \"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1\": rpc error: code = NotFound desc = could not find container \"b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1\": container with ID starting with b26ed915370082523a97a97f41374f1c5b324fd7501cd0b976afd5353f59a0d1 not found: ID does not exist"
Oct 30 19:57:15.464157 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.464127    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:57:15.510928 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:15.510853    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-4vfdh"]
Oct 30 19:57:16.647802 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:16.647754    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" path="/var/lib/kubelet/pods/82afc92c-3c63-42d8-88cc-39b82e11f0f2/volumes"
Oct 30 19:57:18.562724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:18.562655    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"]
Oct 30 19:57:18.594411 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:18.594381    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-99cn2"]
Oct 30 19:57:18.631941 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:18.631911    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2925bca0-51ce-4256-91b4-e205ba1f9a8b" path="/var/lib/kubelet/pods/2925bca0-51ce-4256-91b4-e205ba1f9a8b/volumes"
Oct 30 19:57:19.940587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:19.927757    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"]
Oct 30 19:57:19.972500 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:19.972466    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4477e2-tflgn"]
Oct 30 19:57:20.632651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:20.632622    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b" path="/var/lib/kubelet/pods/f7f7dfd4-843d-4e53-8e1e-cde74d8ae38b/volumes"
Oct 30 19:57:21.238332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:21.238294    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"]
Oct 30 19:57:21.286534 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:21.286497    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61d11a-5hmb4"]
Oct 30 19:57:21.939647 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:21.939608    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:57:22.631774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:22.631733    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b07d55-688f-4e2c-976e-81917a916367" path="/var/lib/kubelet/pods/97b07d55-688f-4e2c-976e-81917a916367/volumes"
Oct 30 19:57:26.991677 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:26.991644    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:57:26.992066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:26.991861    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy" containerID="cri-o://c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7" gracePeriod=30
Oct 30 19:57:29.004843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.004811    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:57:29.005221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.005025    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" containerID="cri-o://92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd" gracePeriod=30
Oct 30 19:57:29.836991 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.836954    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"]
Oct 30 19:57:29.837587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837553    5085 cpu_manager.go:414] "containerMap: removing container" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy"
Oct 30 19:57:29.837587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837581    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy"
Oct 30 19:57:29.837680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837599    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:29.837680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837605    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7bb5ea7-61af-488a-b80b-89c213321588" containerName="istio-proxy"
Oct 30 19:57:29.837680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837622    5085 cpu_manager.go:414] "containerMap: removing container" podUID="607ac483-bdbd-45f1-b159-04fb3e2b2267" containerName="revoke-keys"
Oct 30 19:57:29.837680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837627    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607ac483-bdbd-45f1-b159-04fb3e2b2267" containerName="revoke-keys"
Oct 30 19:57:29.837804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837720    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="607ac483-bdbd-45f1-b159-04fb3e2b2267" containerName="revoke-keys"
Oct 30 19:57:29.837804 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.837744    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="82afc92c-3c63-42d8-88cc-39b82e11f0f2" containerName="istio-proxy"
Oct 30 19:57:29.842548 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.842527    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.865612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.865581    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.865734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.865720    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69nn9\" (UniqueName: \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.865782 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.865768    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.938893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.938852    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:29.938893 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.938886    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:57:29.944859 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.944829    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"]
Oct 30 19:57:29.966551 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.966523    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.966732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.966643    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69nn9\" (UniqueName: \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.966732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.966691    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.967337 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.967316    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:29.968938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:29.968917    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:30.025106 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:30.025068    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69nn9\" (UniqueName: \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\") pod \"maas-api-revoke-keys-e2e-api-b-e67696-j2v9q\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:30.156465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:30.156388    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:30.531905 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:30.531851    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"]
Oct 30 19:57:30.532242 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:57:30.532219    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3b46b9_dad1_4675_8bdf_604252816a47.slice/crio-e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced WatchSource:0}: Error finding container e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced: Status 404 returned error can't find the container with id e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced
Oct 30 19:57:31.406926 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:31.406892    5085 generic.go:356] "Generic (PLEG): container finished" podID="8e3b46b9-dad1-4675-8bdf-604252816a47" containerID="e18308e9fde3694738057a492c482b65353f03d5b08e74260bd4e275328a5339" exitCode=0
Oct 30 19:57:31.407305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:31.406975    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q" event={"ID":"8e3b46b9-dad1-4675-8bdf-604252816a47","Type":"ContainerDied","Data":"e18308e9fde3694738057a492c482b65353f03d5b08e74260bd4e275328a5339"}
Oct 30 19:57:31.407305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:31.407034    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q" event={"ID":"8e3b46b9-dad1-4675-8bdf-604252816a47","Type":"ContainerStarted","Data":"e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced"}
Oct 30 19:57:32.272940 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.272876    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:57:32.292305 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292274    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292320    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292349    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292431 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292370    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292487    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292545    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292610    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292702 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292662    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.292812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292705    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\") pod \"4af2d63d-f72b-4cf3-904d-b158b53c7167\" (UID: \"4af2d63d-f72b-4cf3-904d-b158b53c7167\") "
Oct 30 19:57:32.293141 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.292704    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:32.293141 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293036    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:32.293141 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293054    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:32.293390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293210    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:32.293390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293289    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:32.293390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293346    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.293390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293370    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.293390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293385    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.293626 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.293401    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.294612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.294580    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:32.294843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.294824    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:32.295014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.294994    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "kube-api-access-b4lq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:32.295079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.295023    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token" pod "4af2d63d-f72b-4cf3-904d-b158b53c7167" (UID: "4af2d63d-f72b-4cf3-904d-b158b53c7167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:32.394727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.394697    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.394727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.394723    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4af2d63d-f72b-4cf3-904d-b158b53c7167-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.394727 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.394732    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.394965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.394742    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.394965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.394751    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4lq9\" (UniqueName: \"kubernetes.io/projected/4af2d63d-f72b-4cf3-904d-b158b53c7167-kube-api-access-b4lq9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.418207 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.418172    5085 generic.go:356] "Generic (PLEG): container finished" podID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerID="c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7" exitCode=0
Oct 30 19:57:32.418745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.418249    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"
Oct 30 19:57:32.418745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.418257    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerDied","Data":"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"}
Oct 30 19:57:32.418745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.418314    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq" event={"ID":"4af2d63d-f72b-4cf3-904d-b158b53c7167","Type":"ContainerDied","Data":"4430fa3983ba93b4ea178a944ee217919a7ad781643d37e55747b758bd78613f"}
Oct 30 19:57:32.418745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.418342    5085 scope.go:122] "RemoveContainer" containerID="c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"
Oct 30 19:57:32.434274 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.434245    5085 scope.go:122] "RemoveContainer" containerID="3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"
Oct 30 19:57:32.451139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.451117    5085 scope.go:122] "RemoveContainer" containerID="c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"
Oct 30 19:57:32.451464 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:32.451440    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7\": container with ID starting with c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7 not found: ID does not exist" containerID="c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"
Oct 30 19:57:32.451597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.451478    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7"} err="failed to get container status \"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7\": rpc error: code = NotFound desc = could not find container \"c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7\": container with ID starting with c0081a16e209b005755573825538f15157caa4faa8c64812fb2e92c0afd8e7b7 not found: ID does not exist"
Oct 30 19:57:32.451597 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.451507    5085 scope.go:122] "RemoveContainer" containerID="3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"
Oct 30 19:57:32.451882 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:32.451861    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138\": container with ID starting with 3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138 not found: ID does not exist" containerID="3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"
Oct 30 19:57:32.451957 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.451894    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138"} err="failed to get container status \"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138\": rpc error: code = NotFound desc = could not find container \"3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138\": container with ID starting with 3ccd12d06bd461c9c8159411e6ae869b09134c0996a81f504afc8099a1cc5138 not found: ID does not exist"
Oct 30 19:57:32.563272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.563248    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:32.581715 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.581684    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:57:32.589380 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.589348    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4477e2-openshift-default-57d45d7cc7-hhtdq"]
Oct 30 19:57:32.596236 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.596211    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\") pod \"8e3b46b9-dad1-4675-8bdf-604252816a47\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") "
Oct 30 19:57:32.596393 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.596313    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\") pod \"8e3b46b9-dad1-4675-8bdf-604252816a47\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") "
Oct 30 19:57:32.596433 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.596415    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\" (UniqueName: \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\") pod \"8e3b46b9-dad1-4675-8bdf-604252816a47\" (UID: \"8e3b46b9-dad1-4675-8bdf-604252816a47\") "
Oct 30 19:57:32.596579 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.596541    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca" pod "8e3b46b9-dad1-4675-8bdf-604252816a47" (UID: "8e3b46b9-dad1-4675-8bdf-604252816a47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:32.596858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.596838    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.598448 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.598426    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca" pod "8e3b46b9-dad1-4675-8bdf-604252816a47" (UID: "8e3b46b9-dad1-4675-8bdf-604252816a47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:32.598529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.598474    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9" pod "8e3b46b9-dad1-4675-8bdf-604252816a47" (UID: "8e3b46b9-dad1-4675-8bdf-604252816a47"). InnerVolumeSpecName "kube-api-access-69nn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:32.634602 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.634557    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" path="/var/lib/kubelet/pods/4af2d63d-f72b-4cf3-904d-b158b53c7167/volumes"
Oct 30 19:57:32.698655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.698323    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e3b46b9-dad1-4675-8bdf-604252816a47-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.698655 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.698360    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69nn9\" (UniqueName: \"kubernetes.io/projected/8e3b46b9-dad1-4675-8bdf-604252816a47-kube-api-access-69nn9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:32.892194 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.892113    5085 patch_prober.go:29] interesting pod/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:57:32.892346 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:32.892188    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:57:33.354418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.354387    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:57:33.354989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.354973    5085 cpu_manager.go:414] "containerMap: removing container" podUID="8e3b46b9-dad1-4675-8bdf-604252816a47" containerName="revoke-keys"
Oct 30 19:57:33.355044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.354990    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3b46b9-dad1-4675-8bdf-604252816a47" containerName="revoke-keys"
Oct 30 19:57:33.355044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355025    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.355044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355030    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.355044 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355041    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.355165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355046    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.355165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355128    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3b46b9-dad1-4675-8bdf-604252816a47" containerName="revoke-keys"
Oct 30 19:57:33.355165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355140    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.355165 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.355146    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af2d63d-f72b-4cf3-904d-b158b53c7167" containerName="istio-proxy"
Oct 30 19:57:33.358864 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.358841    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.406260 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.406223    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.406407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.406387    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6h9\" (UniqueName: \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.412708 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.412685    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-111f9c64\""
Oct 30 19:57:33.416319 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.416296    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:57:33.428549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.428525    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"
Oct 30 19:57:33.428951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.428630    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q" event={"ID":"8e3b46b9-dad1-4675-8bdf-604252816a47","Type":"ContainerDied","Data":"e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced"}
Oct 30 19:57:33.428951 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.428711    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a9821f0905984450ff44240282de3fe91313a503acb5442ae77b1595b7eced"
Oct 30 19:57:33.514957 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.507873    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.514957 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.508076    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6h9\" (UniqueName: \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.514957 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.509516    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.537627 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.521273    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:57:33.555651 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.548972    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.575609 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.575267    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6h9\" (UniqueName: \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\") pod \"payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.590034 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.590003    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:57:33.604917 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.604841    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-111f9c64-dockercfg-5r7xv\""
Oct 30 19:57:33.609612 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.609552    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5sz\" (UniqueName: \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.609769 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.609688    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.675169 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.675138    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:33.711105 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.711075    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5sz\" (UniqueName: \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.711266 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.711221    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.712074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.712048    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.773734 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.773697    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5sz\" (UniqueName: \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\") pod \"payload-processing-e2e-mt-111f9c64-759998544b-zkrvk\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") " pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:33.878101 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:33.878016    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:34.055698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.055670    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:57:34.092720 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:57:34.092681    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202d5a84_a865_47e6_a245_f30d3a91c17d.slice/crio-26cb19e2bb72232101273c279d60a8e0c93684054d1a355a1023015e0e6b24cd WatchSource:0}: Error finding container 26cb19e2bb72232101273c279d60a8e0c93684054d1a355a1023015e0e6b24cd: Status 404 returned error can't find the container with id 26cb19e2bb72232101273c279d60a8e0c93684054d1a355a1023015e0e6b24cd
Oct 30 19:57:34.253515 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.253489    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:57:34.256374 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:57:34.256343    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda028d85a_702d_4ecc_ab04_0d4139776c05.slice/crio-89a4d2283c603a0b36e7251138354af423c9cc38fa54b19fe9a58f5c011f189a WatchSource:0}: Error finding container 89a4d2283c603a0b36e7251138354af423c9cc38fa54b19fe9a58f5c011f189a: Status 404 returned error can't find the container with id 89a4d2283c603a0b36e7251138354af423c9cc38fa54b19fe9a58f5c011f189a
Oct 30 19:57:34.272143 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.272119    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:57:34.318607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318578    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318624    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318679    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318733 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318719    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318805    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318842    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.318930 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318924    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.319098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318966    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:34.319098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.318973    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.319251 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.319103    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\") pod \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\" (UID: \"321f5c68-f4f7-4c12-87bb-d10f29ed3517\") "
Oct 30 19:57:34.319322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.319288    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.319859    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.319892    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.319949    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.320008    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.320110    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.321065    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.321357    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.321513    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:34.322604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.321734    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95" pod "321f5c68-f4f7-4c12-87bb-d10f29ed3517" (UID: "321f5c68-f4f7-4c12-87bb-d10f29ed3517"). InnerVolumeSpecName "kube-api-access-dfw95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:34.421201 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421173    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421201 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421200    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421210    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421220    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfw95\" (UniqueName: \"kubernetes.io/projected/321f5c68-f4f7-4c12-87bb-d10f29ed3517-kube-api-access-dfw95\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421232    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421242    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.421443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.421250    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321f5c68-f4f7-4c12-87bb-d10f29ed3517-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:34.441396 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.441362    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" event={"ID":"a028d85a-702d-4ecc-ab04-0d4139776c05","Type":"ContainerStarted","Data":"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"}
Oct 30 19:57:34.441843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.441420    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" event={"ID":"a028d85a-702d-4ecc-ab04-0d4139776c05","Type":"ContainerStarted","Data":"89a4d2283c603a0b36e7251138354af423c9cc38fa54b19fe9a58f5c011f189a"}
Oct 30 19:57:34.441843 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.441554    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:34.443154 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.443129    5085 generic.go:356] "Generic (PLEG): container finished" podID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerID="92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd" exitCode=0
Oct 30 19:57:34.443252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.443166    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"
Oct 30 19:57:34.443252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.443210    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerDied","Data":"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"}
Oct 30 19:57:34.443252 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.443250    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7" event={"ID":"321f5c68-f4f7-4c12-87bb-d10f29ed3517","Type":"ContainerDied","Data":"e0168c317827a75fd12397cc845fa991d70470ec30a2ca2c3e9227dbb29244c5"}
Oct 30 19:57:34.443365 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.443272    5085 scope.go:122] "RemoveContainer" containerID="92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"
Oct 30 19:57:34.445173 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.445146    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" event={"ID":"202d5a84-a865-47e6-a245-f30d3a91c17d","Type":"ContainerStarted","Data":"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"}
Oct 30 19:57:34.445267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.445199    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" event={"ID":"202d5a84-a865-47e6-a245-f30d3a91c17d","Type":"ContainerStarted","Data":"26cb19e2bb72232101273c279d60a8e0c93684054d1a355a1023015e0e6b24cd"}
Oct 30 19:57:34.445267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.445232    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:34.454331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.454316    5085 scope.go:122] "RemoveContainer" containerID="2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"
Oct 30 19:57:34.463788 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.463771    5085 scope.go:122] "RemoveContainer" containerID="92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"
Oct 30 19:57:34.464040 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:34.464020    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd\": container with ID starting with 92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd not found: ID does not exist" containerID="92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"
Oct 30 19:57:34.464111 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.464051    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd"} err="failed to get container status \"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd\": rpc error: code = NotFound desc = could not find container \"92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd\": container with ID starting with 92dced22b8752ffad301a28af9b180dc6114c862a15db663d3d6ca983ed044fd not found: ID does not exist"
Oct 30 19:57:34.464111 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.464077    5085 scope.go:122] "RemoveContainer" containerID="2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"
Oct 30 19:57:34.464330 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:34.464311    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2\": container with ID starting with 2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2 not found: ID does not exist" containerID="2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"
Oct 30 19:57:34.464366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.464337    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2"} err="failed to get container status \"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2\": rpc error: code = NotFound desc = could not find container \"2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2\": container with ID starting with 2b0c010d2c8e60db6c936cc63847f81a3b6282f2309c2f2a5bb711af4721d2c2 not found: ID does not exist"
Oct 30 19:57:34.552381 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.552332    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" podStartSLOduration=1.552317124 podStartE2EDuration="1.552317124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:34.550741576 +0000 UTC m=+1964.055145323" watchObservedRunningTime="2026-10-30 19:57:34.552317124 +0000 UTC m=+1964.056720869"
Oct 30 19:57:34.676375 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.673504    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" podStartSLOduration=1.6734826329999999 podStartE2EDuration="1.673482633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:34.660550064 +0000 UTC m=+1964.164953811" watchObservedRunningTime="2026-10-30 19:57:34.673482633 +0000 UTC m=+1964.177886381"
Oct 30 19:57:34.769024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.768989    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:57:34.774781 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:34.774748    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-61d11a-openshift-default-cdfc559c7-k2bs7"]
Oct 30 19:57:36.632400 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:36.632363    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" path="/var/lib/kubelet/pods/321f5c68-f4f7-4c12-87bb-d10f29ed3517/volumes"
Oct 30 19:57:45.457750 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:45.457715    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:57:45.458257 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:45.457975    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:57:51.938728 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:57:51.938689    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:57:53.004884 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:53.004844    5085 scope.go:122] "RemoveContainer" containerID="79ec794eca429a29359ca3f59982cb8c059393a28af42e340bbe3fbb18201c22"
Oct 30 19:57:53.233355 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:53.233304    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:57:54.491318 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:54.491287    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:57:54.491732 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:54.491536    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" containerID="cri-o://73ed190d4775b380fa9544d20809b292b79f8e8d3912fc5e94e6d960043a8beb" gracePeriod=30
Oct 30 19:57:56.508338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.508303    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"]
Oct 30 19:57:56.508896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.508881    5085 cpu_manager.go:414] "containerMap: removing container" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.508944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.508897    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.508944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.508915    5085 cpu_manager.go:414] "containerMap: removing container" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.508944 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.508921    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.509041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.509022    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.509041 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.509033    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="321f5c68-f4f7-4c12-87bb-d10f29ed3517" containerName="istio-proxy"
Oct 30 19:57:56.514440 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.514420    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.567763 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.567735    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"]
Oct 30 19:57:56.599853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.599818    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4lk\" (UniqueName: \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.600025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.599866    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.600025 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.599983    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.610180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.610148    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:57:56.610331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.610237    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:57:56.701272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.701235    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4lk\" (UniqueName: \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.701272 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.701269    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.701488 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.701390    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.702052 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.702024    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.703866 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.703835    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.760217 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.760155    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4lk\" (UniqueName: \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\") pod \"maas-api-revoke-sub-226ab710-8h76h\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:56.828148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:56.828121    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:57.200325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:57.200294    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"]
Oct 30 19:57:57.201600 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:57:57.201544    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3281494f_8406_4ff8_8b40_ca1fd3ae6979.slice/crio-08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c WatchSource:0}: Error finding container 08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c: Status 404 returned error can't find the container with id 08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c
Oct 30 19:57:57.622063 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:57.622019    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h" event={"ID":"3281494f-8406-4ff8-8b40-ca1fd3ae6979","Type":"ContainerStarted","Data":"bdf37272df0facab9b9ded6a309fe6f1477ded0ebf38773d701beed13015241a"}
Oct 30 19:57:57.622539 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:57.622083    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h" event={"ID":"3281494f-8406-4ff8-8b40-ca1fd3ae6979","Type":"ContainerStarted","Data":"08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c"}
Oct 30 19:57:57.783765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:57.783668    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h" podStartSLOduration=1.783653478 podStartE2EDuration="1.783653478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:57.735368758 +0000 UTC m=+1987.239772504" watchObservedRunningTime="2026-10-30 19:57:57.783653478 +0000 UTC m=+1987.288057224"
Oct 30 19:57:58.625676 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:58.625636    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-rmnvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:57:58.626170 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:58.625725    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:57:58.631714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:58.631681    5085 generic.go:356] "Generic (PLEG): container finished" podID="3281494f-8406-4ff8-8b40-ca1fd3ae6979" containerID="bdf37272df0facab9b9ded6a309fe6f1477ded0ebf38773d701beed13015241a" exitCode=0
Oct 30 19:57:58.631854 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:58.631801    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h" event={"ID":"3281494f-8406-4ff8-8b40-ca1fd3ae6979","Type":"ContainerDied","Data":"bdf37272df0facab9b9ded6a309fe6f1477ded0ebf38773d701beed13015241a"}
Oct 30 19:57:59.641786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.641754    5085 generic.go:356] "Generic (PLEG): container finished" podID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerID="73ed190d4775b380fa9544d20809b292b79f8e8d3912fc5e94e6d960043a8beb" exitCode=0
Oct 30 19:57:59.642184 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.641823    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" event={"ID":"fae7b2c8-2eed-4727-bb14-d0618dedca38","Type":"ContainerDied","Data":"73ed190d4775b380fa9544d20809b292b79f8e8d3912fc5e94e6d960043a8beb"}
Oct 30 19:57:59.779518 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.779454    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:57:59.824540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.824514    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:57:59.833678 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833654    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.833806 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833786    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.833873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833813    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\" (UniqueName: \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\") pod \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") "
Oct 30 19:57:59.833873 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833855    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.833976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833885    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.833976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833915    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.833976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.833946    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834110    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834184    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834200    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834193    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834285    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834412    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\") pod \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") "
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834500    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:59.834542 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834513    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\") pod \"fae7b2c8-2eed-4727-bb14-d0618dedca38\" (UID: \"fae7b2c8-2eed-4727-bb14-d0618dedca38\") "
Oct 30 19:57:59.835144 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.834617    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\") pod \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\" (UID: \"3281494f-8406-4ff8-8b40-ca1fd3ae6979\") "
Oct 30 19:57:59.835209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835153    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.835209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835175    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.835209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835192    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.835765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835207    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae7b2c8-2eed-4727-bb14-d0618dedca38-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.835765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835548    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca" pod "3281494f-8406-4ff8-8b40-ca1fd3ae6979" (UID: "3281494f-8406-4ff8-8b40-ca1fd3ae6979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:59.835765 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.835711    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:59.836350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836327    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk" pod "3281494f-8406-4ff8-8b40-ca1fd3ae6979" (UID: "3281494f-8406-4ff8-8b40-ca1fd3ae6979"). InnerVolumeSpecName "kube-api-access-qt4lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:59.836350 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836355    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "kube-api-access-c9mn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:59.836508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836411    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:59.836508 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836506    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:57:59.836641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836627    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo" pod "fae7b2c8-2eed-4727-bb14-d0618dedca38" (UID: "fae7b2c8-2eed-4727-bb14-d0618dedca38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:57:59.836753 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.836740    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca" pod "3281494f-8406-4ff8-8b40-ca1fd3ae6979" (UID: "3281494f-8406-4ff8-8b40-ca1fd3ae6979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:59.935892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935857    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.935892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935886    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935900    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae7b2c8-2eed-4727-bb14-d0618dedca38-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935913    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935924    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae7b2c8-2eed-4727-bb14-d0618dedca38-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935935    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3281494f-8406-4ff8-8b40-ca1fd3ae6979-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935949    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mn6\" (UniqueName: \"kubernetes.io/projected/fae7b2c8-2eed-4727-bb14-d0618dedca38-kube-api-access-c9mn6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:57:59.936108 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:57:59.935961    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt4lk\" (UniqueName: \"kubernetes.io/projected/3281494f-8406-4ff8-8b40-ca1fd3ae6979-kube-api-access-qt4lk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:00.649670 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.649647    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"
Oct 30 19:58:00.650042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.649646    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr" event={"ID":"fae7b2c8-2eed-4727-bb14-d0618dedca38","Type":"ContainerDied","Data":"d4e601b1b57f642ee34d4aa5ae139514bd4834597c4712b620ae6c804f92a729"}
Oct 30 19:58:00.650042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.649770    5085 scope.go:122] "RemoveContainer" containerID="73ed190d4775b380fa9544d20809b292b79f8e8d3912fc5e94e6d960043a8beb"
Oct 30 19:58:00.651545 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.651232    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"
Oct 30 19:58:00.651545 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.651273    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h" event={"ID":"3281494f-8406-4ff8-8b40-ca1fd3ae6979","Type":"ContainerDied","Data":"08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c"}
Oct 30 19:58:00.651545 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.651323    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08119f300761fea3a8befc0f0b6d1b068e44fe5f1ed2f77f1299d44292eb821c"
Oct 30 19:58:00.912803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.912722    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:58:00.918417 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:00.918390    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-rmnvr"]
Oct 30 19:58:02.617682 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.617652    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:58:02.618298 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618281    5085 cpu_manager.go:414] "containerMap: removing container" podUID="3281494f-8406-4ff8-8b40-ca1fd3ae6979" containerName="revoke-keys"
Oct 30 19:58:02.618348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618300    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3281494f-8406-4ff8-8b40-ca1fd3ae6979" containerName="revoke-keys"
Oct 30 19:58:02.618348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618327    5085 cpu_manager.go:414] "containerMap: removing container" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy"
Oct 30 19:58:02.618348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618331    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy"
Oct 30 19:58:02.618455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618424    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" containerName="istio-proxy"
Oct 30 19:58:02.618455 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.618439    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="3281494f-8406-4ff8-8b40-ca1fd3ae6979" containerName="revoke-keys"
Oct 30 19:58:02.623698 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.623672    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.634066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.634039    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae7b2c8-2eed-4727-bb14-d0618dedca38" path="/var/lib/kubelet/pods/fae7b2c8-2eed-4727-bb14-d0618dedca38/volumes"
Oct 30 19:58:02.665663 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.665630    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws462\" (UniqueName: \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.665809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.665680    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.665809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.665766    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.665971 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.665954    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.675686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.675663    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-111f9c64-serving-cert\""
Oct 30 19:58:02.727972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.727938    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:58:02.766906 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.766876    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.767097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.766939    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws462\" (UniqueName: \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.767097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.766977    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.767097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.767006    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.767097 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:02.767055    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-serving-cert: secret "maas-api-e2e-mt-111f9c64-serving-cert" not found
Oct 30 19:58:02.767331 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:02.767116    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls podName:2759869e-8581-47f7-881f-bf4dace05e07 nodeName:}" failed. No retries permitted until 2026-10-30 19:58:03.267097542 +0000 UTC m=+1992.771501284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls") pod "maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" (UID: "2759869e-8581-47f7-881f-bf4dace05e07") : secret "maas-api-e2e-mt-111f9c64-serving-cert" not found
Oct 30 19:58:02.768047 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.768013    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.770556 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.770503    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:02.826275 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:02.826244    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws462\" (UniqueName: \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:03.272581 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:03.272533    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:03.274994 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:03.274966    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") pod \"maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:03.538297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:03.538220    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:03.910392 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:03.910361    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:58:03.911743 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:58:03.911706    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2759869e_8581_47f7_881f_bf4dace05e07.slice/crio-b7877d34d69cc7a4965b3491f5035c5c9972af178748b9964510e2169494a9a7 WatchSource:0}: Error finding container b7877d34d69cc7a4965b3491f5035c5c9972af178748b9964510e2169494a9a7: Status 404 returned error can't find the container with id b7877d34d69cc7a4965b3491f5035c5c9972af178748b9964510e2169494a9a7
Oct 30 19:58:03.913110 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:03.913091    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:58:04.683075 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:04.683038    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" event={"ID":"2759869e-8581-47f7-881f-bf4dace05e07","Type":"ContainerStarted","Data":"b7877d34d69cc7a4965b3491f5035c5c9972af178748b9964510e2169494a9a7"}
Oct 30 19:58:05.691198 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:05.691166    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" event={"ID":"2759869e-8581-47f7-881f-bf4dace05e07","Type":"ContainerStarted","Data":"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"}
Oct 30 19:58:05.691594 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:05.691292    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:05.804603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:05.804520    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" podStartSLOduration=2.9616539299999998 podStartE2EDuration="3.804500344s" totalImagesPullingTime="842.846414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:05.802610905 +0000 UTC m=+1995.307014670" watchObservedRunningTime="2026-10-30 19:58:05.804500344 +0000 UTC m=+1995.308904092"
Oct 30 19:58:11.703767 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:11.703737    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:58:14.913993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:14.913955    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"]
Oct 30 19:58:14.960610 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:14.960577    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4b1a4c-cxltj"]
Oct 30 19:58:16.632308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:16.632273    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cdba21-2590-456e-8d9d-400d60adc640" path="/var/lib/kubelet/pods/32cdba21-2590-456e-8d9d-400d60adc640/volumes"
Oct 30 19:58:21.939731 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:21.939687    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:58:24.848649 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.848611    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 19:58:24.854027 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.854007    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.858863 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.858837    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.858965 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.858910    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859016 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.858995    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859053 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859032    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859091 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859059    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbf2c\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859144    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859199    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859243    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.859308 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.859292    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.880860 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.880838    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 19:58:24.960158 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960125    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960158 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960162    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960182    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960201    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbf2c\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960252    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960285    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960346    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960390 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960370    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960447    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960675    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960724 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960696    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960750    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.960896 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.960857    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.961156 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.961132    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.962855 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.962830    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:24.963084 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:24.963066    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:25.030033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.030000    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:25.030193 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.030130    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbf2c\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\") pod \"maas-default-gateway-openshift-default-694db55c87-p82xp\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:25.167858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.167739    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:25.604841 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.604810    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 19:58:25.605770 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:58:25.605742    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5c97c8_7a5d_4163_ac58_8bcccb4f2777.slice/crio-4b985abc6d09a450314ebdd439226363fdb6b70b520f7bc1f6306b92bcd3e0c9 WatchSource:0}: Error finding container 4b985abc6d09a450314ebdd439226363fdb6b70b520f7bc1f6306b92bcd3e0c9: Status 404 returned error can't find the container with id 4b985abc6d09a450314ebdd439226363fdb6b70b520f7bc1f6306b92bcd3e0c9
Oct 30 19:58:25.608161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.608129    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:25.608244 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.608195    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:25.608244 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.608228    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:25.837684 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.837639    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" event={"ID":"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777","Type":"ContainerStarted","Data":"a058f9502c8d886969fa1af5ddf36459a2a431fe26f0922fbd8e108de0bd9efd"}
Oct 30 19:58:25.837854 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.837710    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" event={"ID":"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777","Type":"ContainerStarted","Data":"4b985abc6d09a450314ebdd439226363fdb6b70b520f7bc1f6306b92bcd3e0c9"}
Oct 30 19:58:25.955273 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:25.955212    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podStartSLOduration=1.955197111 podStartE2EDuration="1.955197111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:25.952363265 +0000 UTC m=+2015.456767048" watchObservedRunningTime="2026-10-30 19:58:25.955197111 +0000 UTC m=+2015.459600857"
Oct 30 19:58:26.168212 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:26.168175    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:26.169645 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:26.169621    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.169761 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:26.169667    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:27.171591 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:27.169398    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.185611 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:27.182655    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:27.699767 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:27.686604    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"]
Oct 30 19:58:27.714544 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:27.714511    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-69932e-qjbgp"]
Oct 30 19:58:28.179641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:28.177045    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.179641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:28.177118    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:28.632289 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:28.632209    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607ac483-bdbd-45f1-b159-04fb3e2b2267" path="/var/lib/kubelet/pods/607ac483-bdbd-45f1-b159-04fb3e2b2267/volumes"
Oct 30 19:58:29.176989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:29.176951    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.177284 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:29.177255    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:30.178583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:30.172974    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.178583 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:30.173035    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:31.177024 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:31.176985    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.177313 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:31.177283    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:31.255338 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:31.255309    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:58:32.168766 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:32.168733    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:32.168981 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:32.168791    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:33.168489 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:33.168451    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-p82xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 19:58:33.168891 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:33.168505    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 19:58:34.520434 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:34.520401    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:34.520949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:34.520764    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:34.521694 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:34.521674    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 19:58:36.506740 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.506705    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"]
Oct 30 19:58:36.512680 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.512653    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.569113 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.569076    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"]
Oct 30 19:58:36.609797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.609763    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:58:36.609797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.609791    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:58:36.695483 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.695443    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.695745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.695499    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.695745 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.695710    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2lrc\" (UniqueName: \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.797149 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.797059    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2lrc\" (UniqueName: \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.797334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.797168    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.797334 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.797209    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.798226 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.798201    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.799950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.799926    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:36.855936 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:36.855908    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2lrc\" (UniqueName: \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\") pod \"maas-api-revoke-keys-e2e-api-b-585635-wzvzr\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:37.129020 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:37.128949    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:37.504879 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:37.504848    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"]
Oct 30 19:58:37.505777 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:58:37.505738    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e84f5eb_004b_4cbb_994d_b0632b4d4889.slice/crio-a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5 WatchSource:0}: Error finding container a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5: Status 404 returned error can't find the container with id a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5
Oct 30 19:58:37.959623 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:37.959554    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr" event={"ID":"1e84f5eb-004b-4cbb-994d-b0632b4d4889","Type":"ContainerStarted","Data":"10f170354159009544035c04078974438d08074f595e4f41745561657f598247"}
Oct 30 19:58:37.960022 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:37.959650    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr" event={"ID":"1e84f5eb-004b-4cbb-994d-b0632b4d4889","Type":"ContainerStarted","Data":"a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5"}
Oct 30 19:58:38.070415 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:38.070309    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr" podStartSLOduration=2.070290609 podStartE2EDuration="2.070290609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:38.069999214 +0000 UTC m=+2027.574402960" watchObservedRunningTime="2026-10-30 19:58:38.070290609 +0000 UTC m=+2027.574694360"
Oct 30 19:58:38.968130 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:38.968095    5085 generic.go:356] "Generic (PLEG): container finished" podID="1e84f5eb-004b-4cbb-994d-b0632b4d4889" containerID="10f170354159009544035c04078974438d08074f595e4f41745561657f598247" exitCode=0
Oct 30 19:58:38.968547 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:38.968138    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr" event={"ID":"1e84f5eb-004b-4cbb-994d-b0632b4d4889","Type":"ContainerDied","Data":"10f170354159009544035c04078974438d08074f595e4f41745561657f598247"}
Oct 30 19:58:39.968704 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:39.968661    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 19:58:39.974669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:39.974636    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036216 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036185    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036386 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036249    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036425    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsbjt\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036581    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036644 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036611    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036700    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036774 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036750    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.036912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.036889    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.037042 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.037025    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.092603 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.092557    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 19:58:40.145232 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145203    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145244    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145278    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145312    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsbjt\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145360    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145376    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145420 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145409    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145783 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145447    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145783 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145511    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.145973 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.145897    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.146509 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.146484    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.146858 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.146837    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.147032 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.147011    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.147249 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.147222    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.148938 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.148912    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.149172 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.149151    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.177425 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.177403    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:40.204931 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.204906    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.205098 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.205077    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsbjt\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\") pod \"maas-default-gateway-openshift-default-694db55c87-fpsfd\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.246803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.246723    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\") pod \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") "
Oct 30 19:58:40.246923 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.246821    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\" (UniqueName: \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\") pod \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") "
Oct 30 19:58:40.246972 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.246931    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\") pod \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\" (UID: \"1e84f5eb-004b-4cbb-994d-b0632b4d4889\") "
Oct 30 19:58:40.247288 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.247261    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca" pod "1e84f5eb-004b-4cbb-994d-b0632b4d4889" (UID: "1e84f5eb-004b-4cbb-994d-b0632b4d4889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:40.247435 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.247419    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.248749 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.248725    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc" pod "1e84f5eb-004b-4cbb-994d-b0632b4d4889" (UID: "1e84f5eb-004b-4cbb-994d-b0632b4d4889"). InnerVolumeSpecName "kube-api-access-l2lrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:40.248850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.248786    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca" pod "1e84f5eb-004b-4cbb-994d-b0632b4d4889" (UID: "1e84f5eb-004b-4cbb-994d-b0632b4d4889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:40.289797 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.289769    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:40.349269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.349239    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e84f5eb-004b-4cbb-994d-b0632b4d4889-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.349269 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.349270    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2lrc\" (UniqueName: \"kubernetes.io/projected/1e84f5eb-004b-4cbb-994d-b0632b4d4889-kube-api-access-l2lrc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:40.719587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.719545    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 19:58:40.720463 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:58:40.720438    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cee3d9_2472_45a8_adb9_11a876d6ea21.slice/crio-27a7efc74fdbaa474fa7ed9939bc6f385dc2534fcca5bdb7d1fc58d9515affc8 WatchSource:0}: Error finding container 27a7efc74fdbaa474fa7ed9939bc6f385dc2534fcca5bdb7d1fc58d9515affc8: Status 404 returned error can't find the container with id 27a7efc74fdbaa474fa7ed9939bc6f385dc2534fcca5bdb7d1fc58d9515affc8
Oct 30 19:58:40.723007 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.722972    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:40.723114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.723051    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:40.723114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.723099    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 19:58:40.988322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.988287    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"
Oct 30 19:58:40.988759 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.988328    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr" event={"ID":"1e84f5eb-004b-4cbb-994d-b0632b4d4889","Type":"ContainerDied","Data":"a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5"}
Oct 30 19:58:40.988759 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.988375    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59b3ef08d3218a985bbf2c9513a03e5e2aeabed6af8cb13dbce7359909b5da5"
Oct 30 19:58:40.990501 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.990469    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" event={"ID":"89cee3d9-2472-45a8-adb9-11a876d6ea21","Type":"ContainerStarted","Data":"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"}
Oct 30 19:58:40.990631 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:40.990525    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" event={"ID":"89cee3d9-2472-45a8-adb9-11a876d6ea21","Type":"ContainerStarted","Data":"27a7efc74fdbaa474fa7ed9939bc6f385dc2534fcca5bdb7d1fc58d9515affc8"}
Oct 30 19:58:41.107954 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:41.107895    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podStartSLOduration=2.107881954 podStartE2EDuration="2.107881954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:41.105878366 +0000 UTC m=+2030.610282114" watchObservedRunningTime="2026-10-30 19:58:41.107881954 +0000 UTC m=+2030.612285700"
Oct 30 19:58:41.290643 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:41.290536    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:41.292177 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:41.292150    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:41.292315 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:41.292198    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:42.290352 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:42.290310    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:42.290800 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:42.290374    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:43.290209 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:43.290168    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:43.290389 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:43.290240    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:44.290803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:44.290767    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:44.291258 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:44.290830    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:45.290822 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:45.290781    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:45.291310 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:45.290855    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:46.290276 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:46.290239    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:46.290456 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:46.290322    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:47.291361 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:47.291318    5085 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-694db55c87-fpsfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 30 19:58:47.291786 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:47.291373    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 30 19:58:48.068615 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:48.068578    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:58:48.171531 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:48.171492    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:58:49.134331 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:49.134273    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:49.135332 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:49.135108    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:49.136726 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:49.136699    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 19:58:51.939597 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:51.939546    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:58:53.069912 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.069870    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" podUID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" containerName="payload-processing" containerID="cri-o://986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec" gracePeriod=25
Oct 30 19:58:53.172606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.172549    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" podUID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" containerName="payload-pre-processing" containerID="cri-o://b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c" gracePeriod=25
Oct 30 19:58:53.356898 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.356522    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:58:53.447916 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.447892    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\" (UniqueName: \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\") pod \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") "
Oct 30 19:58:53.448034 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.448017    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\") pod \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\" (UID: \"ca0ee7a3-eb42-4c72-913e-1efccd4fe559\") "
Oct 30 19:58:53.448374 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.448350    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume" pod "ca0ee7a3-eb42-4c72-913e-1efccd4fe559" (UID: "ca0ee7a3-eb42-4c72-913e-1efccd4fe559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:53.448550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.448537    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:53.449975 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.449951    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg" pod "ca0ee7a3-eb42-4c72-913e-1efccd4fe559" (UID: "ca0ee7a3-eb42-4c72-913e-1efccd4fe559"). InnerVolumeSpecName "kube-api-access-p64kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:53.457949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.457929    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:58:53.549771 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.549741    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\") pod \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") "
Oct 30 19:58:53.549950 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.549936    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\" (UniqueName: \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\") pod \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\" (UID: \"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72\") "
Oct 30 19:58:53.550116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.550091    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume" pod "90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" (UID: "90dd14bc-ef0f-49e4-ad7a-2ac957aaed72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:53.550256 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.550243    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64kg\" (UniqueName: \"kubernetes.io/projected/ca0ee7a3-eb42-4c72-913e-1efccd4fe559-kube-api-access-p64kg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:53.550329 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.550258    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:53.552014 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.551991    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls" pod "90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" (UID: "90dd14bc-ef0f-49e4-ad7a-2ac957aaed72"). InnerVolumeSpecName "kube-api-access-m25ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:53.651463 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:53.651388    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25ls\" (UniqueName: \"kubernetes.io/projected/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72-kube-api-access-m25ls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:58:54.082342 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.082304    5085 generic.go:356] "Generic (PLEG): container finished" podID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" containerID="b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c" exitCode=0
Oct 30 19:58:54.082853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.082354    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"
Oct 30 19:58:54.082853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.082391    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" event={"ID":"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72","Type":"ContainerDied","Data":"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"}
Oct 30 19:58:54.082853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.082463    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5" event={"ID":"90dd14bc-ef0f-49e4-ad7a-2ac957aaed72","Type":"ContainerDied","Data":"3cd407f9831642899df5167c5fbb8ecd5968f93dc2dcd4a784552e8c25310204"}
Oct 30 19:58:54.082853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.082499    5085 scope.go:122] "RemoveContainer" containerID="b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"
Oct 30 19:58:54.084088 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.084063    5085 generic.go:356] "Generic (PLEG): container finished" podID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" containerID="986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec" exitCode=0
Oct 30 19:58:54.084188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.084120    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"
Oct 30 19:58:54.084188 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.084125    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" event={"ID":"ca0ee7a3-eb42-4c72-913e-1efccd4fe559","Type":"ContainerDied","Data":"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"}
Oct 30 19:58:54.084301 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.084232    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982" event={"ID":"ca0ee7a3-eb42-4c72-913e-1efccd4fe559","Type":"ContainerDied","Data":"1d58d5b1053643a3451a48002efd0a212bcce155fecee979fb2c9744a9e4a208"}
Oct 30 19:58:54.098003 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.097985    5085 scope.go:122] "RemoveContainer" containerID="b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"
Oct 30 19:58:54.098301 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:54.098282    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c\": container with ID starting with b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c not found: ID does not exist" containerID="b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"
Oct 30 19:58:54.098344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.098308    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c"} err="failed to get container status \"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c\": rpc error: code = NotFound desc = could not find container \"b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c\": container with ID starting with b55b9b0d391a121e1326999b1884e70e902cd39875e53177c2de5a3f4101cd2c not found: ID does not exist"
Oct 30 19:58:54.098344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.098326    5085 scope.go:122] "RemoveContainer" containerID="986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"
Oct 30 19:58:54.108949 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.108932    5085 scope.go:122] "RemoveContainer" containerID="986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"
Oct 30 19:58:54.109200 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:58:54.109182    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec\": container with ID starting with 986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec not found: ID does not exist" containerID="986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"
Oct 30 19:58:54.109248 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.109209    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec"} err="failed to get container status \"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec\": rpc error: code = NotFound desc = could not find container \"986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec\": container with ID starting with 986558968a92f91c26ba1dd3bb9dd824b68da94322a55dc601242f3ac3899fec not found: ID does not exist"
Oct 30 19:58:54.243809 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.243769    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:58:54.288686 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.288656    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-585635-5dffcfd79-d2dg5"]
Oct 30 19:58:54.352867 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.352791    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:58:54.359277 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.359231    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-585635-5598c7d4d-f4982"]
Oct 30 19:58:54.633947 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.633875    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" path="/var/lib/kubelet/pods/90dd14bc-ef0f-49e4-ad7a-2ac957aaed72/volumes"
Oct 30 19:58:54.634344 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:58:54.634329    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" path="/var/lib/kubelet/pods/ca0ee7a3-eb42-4c72-913e-1efccd4fe559/volumes"
Oct 30 19:59:01.817267 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:01.817231    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"]
Oct 30 19:59:01.863955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:01.863917    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e67696-j2v9q"]
Oct 30 19:59:02.631407 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:02.631373    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3b46b9-dad1-4675-8bdf-604252816a47" path="/var/lib/kubelet/pods/8e3b46b9-dad1-4675-8bdf-604252816a47/volumes"
Oct 30 19:59:06.292989 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.292952    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"]
Oct 30 19:59:06.293607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293591    5085 cpu_manager.go:414] "containerMap: removing container" podUID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" containerName="payload-pre-processing"
Oct 30 19:59:06.293664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293608    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" containerName="payload-pre-processing"
Oct 30 19:59:06.293664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293636    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" containerName="payload-processing"
Oct 30 19:59:06.293664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293641    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" containerName="payload-processing"
Oct 30 19:59:06.293664 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293660    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1e84f5eb-004b-4cbb-994d-b0632b4d4889" containerName="revoke-keys"
Oct 30 19:59:06.293790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293665    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e84f5eb-004b-4cbb-994d-b0632b4d4889" containerName="revoke-keys"
Oct 30 19:59:06.293790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293748    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="90dd14bc-ef0f-49e4-ad7a-2ac957aaed72" containerName="payload-pre-processing"
Oct 30 19:59:06.293790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293759    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0ee7a3-eb42-4c72-913e-1efccd4fe559" containerName="payload-processing"
Oct 30 19:59:06.293790 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.293769    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e84f5eb-004b-4cbb-994d-b0632b4d4889" containerName="revoke-keys"
Oct 30 19:59:06.298929 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.298909    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.303836 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.303811    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgn6g\" (UniqueName: \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.303937 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.303846    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.303978 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.303952    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.353106 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.353077    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"]
Oct 30 19:59:06.395770 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.395739    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 19:59:06.395911 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.395775    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 19:59:06.404955 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.404930    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgn6g\" (UniqueName: \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.405074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.404966    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.405074 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.405032    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.405633 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.405602    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.407325 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.407306    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.463980 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.463951    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgn6g\" (UniqueName: \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\") pod \"maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.611798 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.611721    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:06.996987 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:06.996955    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"]
Oct 30 19:59:06.998320 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:59:06.998288    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5b8471_7580_494f_be07_a64b2e5dae22.slice/crio-6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0 WatchSource:0}: Error finding container 6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0: Status 404 returned error can't find the container with id 6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0
Oct 30 19:59:07.194140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:07.194104    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf" event={"ID":"bc5b8471-7580-494f-be07-a64b2e5dae22","Type":"ContainerStarted","Data":"6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0"}
Oct 30 19:59:08.203604 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:08.203511    5085 generic.go:356] "Generic (PLEG): container finished" podID="bc5b8471-7580-494f-be07-a64b2e5dae22" containerID="7984916013f710e20eae9536e29f021b8942dfc46547124a73375adfcef8d95a" exitCode=0
Oct 30 19:59:08.203984 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:08.203598    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf" event={"ID":"bc5b8471-7580-494f-be07-a64b2e5dae22","Type":"ContainerDied","Data":"7984916013f710e20eae9536e29f021b8942dfc46547124a73375adfcef8d95a"}
Oct 30 19:59:09.172079 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.172048    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:59:09.172271 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.172247    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy" containerID="cri-o://9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3" gracePeriod=30
Oct 30 19:59:09.449868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.449837    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:09.535161 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.535134    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\") pod \"bc5b8471-7580-494f-be07-a64b2e5dae22\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") "
Oct 30 19:59:09.535354 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.535286    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\" (UniqueName: \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\") pod \"bc5b8471-7580-494f-be07-a64b2e5dae22\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") "
Oct 30 19:59:09.535424 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.535366    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\") pod \"bc5b8471-7580-494f-be07-a64b2e5dae22\" (UID: \"bc5b8471-7580-494f-be07-a64b2e5dae22\") "
Oct 30 19:59:09.535803 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.535775    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca" pod "bc5b8471-7580-494f-be07-a64b2e5dae22" (UID: "bc5b8471-7580-494f-be07-a64b2e5dae22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.537416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.537386    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g" pod "bc5b8471-7580-494f-be07-a64b2e5dae22" (UID: "bc5b8471-7580-494f-be07-a64b2e5dae22"). InnerVolumeSpecName "kube-api-access-kgn6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:09.537416 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.537387    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca" pod "bc5b8471-7580-494f-be07-a64b2e5dae22" (UID: "bc5b8471-7580-494f-be07-a64b2e5dae22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:09.637546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.637501    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.637546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.637537    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5b8471-7580-494f-be07-a64b2e5dae22-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.637546 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:09.637550    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgn6g\" (UniqueName: \"kubernetes.io/projected/bc5b8471-7580-494f-be07-a64b2e5dae22-kube-api-access-kgn6g\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.218296 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:10.218270    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"
Oct 30 19:59:10.218465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:10.218318    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf" event={"ID":"bc5b8471-7580-494f-be07-a64b2e5dae22","Type":"ContainerDied","Data":"6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0"}
Oct 30 19:59:10.218465 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:10.218369    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6694475fdecb1d48a04300e23fe2fb413188d2eff57b49d1202c153e24127eb0"
Oct 30 19:59:12.160529 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.160495    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 19:59:12.161076 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.161062    5085 cpu_manager.go:414] "containerMap: removing container" podUID="bc5b8471-7580-494f-be07-a64b2e5dae22" containerName="revoke-keys"
Oct 30 19:59:12.161124 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.161078    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5b8471-7580-494f-be07-a64b2e5dae22" containerName="revoke-keys"
Oct 30 19:59:12.161241 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.161230    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5b8471-7580-494f-be07-a64b2e5dae22" containerName="revoke-keys"
Oct 30 19:59:12.164796 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.164777    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.263507 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.263476    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d80c7d-serving-cert\""
Oct 30 19:59:12.269097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.269066    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 19:59:12.277322 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.277288    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mw7w\" (UniqueName: \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.277550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.277439    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.277550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.277530    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.277720 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.277556    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.398459 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.398420    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mw7w\" (UniqueName: \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.398695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.398545    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.398695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.398654    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.398695 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.398681    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.399672 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.399648    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.400391 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:12.400367    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-serving-cert: secret "maas-api-e2e-ipp-a-d80c7d-serving-cert" not found
Oct 30 19:59:12.400628 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:12.400611    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls podName:10939141-efa0-40c6-8bc8-553f16fe3325 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:12.900536742 +0000 UTC m=+2062.404940468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls") pod "maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" (UID: "10939141-efa0-40c6-8bc8-553f16fe3325") : secret "maas-api-e2e-ipp-a-d80c7d-serving-cert" not found
Oct 30 19:59:12.407114 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.407089    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.457714 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.457685    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mw7w\" (UniqueName: \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.905518 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.905441    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:12.907817 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:12.907793    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:13.078510 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:13.078475    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:13.472616 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:13.472458    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 19:59:13.473898 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:59:13.473872    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10939141_efa0_40c6_8bc8_553f16fe3325.slice/crio-082840c208499ec69777bc3adaae42d98c1ed4070c50df387255fe3f7d9bcd7a WatchSource:0}: Error finding container 082840c208499ec69777bc3adaae42d98c1ed4070c50df387255fe3f7d9bcd7a: Status 404 returned error can't find the container with id 082840c208499ec69777bc3adaae42d98c1ed4070c50df387255fe3f7d9bcd7a
Oct 30 19:59:14.256327 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.256288    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" event={"ID":"10939141-efa0-40c6-8bc8-553f16fe3325","Type":"ContainerStarted","Data":"082840c208499ec69777bc3adaae42d98c1ed4070c50df387255fe3f7d9bcd7a"}
Oct 30 19:59:14.477892 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.477868    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:59:14.527428 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527396    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527521    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527646 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527578    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527663    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527673    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:14.527772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527703    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527772 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527750    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527789    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527816    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:14.527979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527834    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.527979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.527882    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\") pod \"51cb27e2-420d-472f-95cf-2d2ea5359485\" (UID: \"51cb27e2-420d-472f-95cf-2d2ea5359485\") "
Oct 30 19:59:14.528180 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528136    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:14.528348 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528328    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:14.528438 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528422    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.528490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528445    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.528490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528456    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.528490 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528465    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.528869 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.528846    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:14.530414 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.530391    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:14.530414 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.530399    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:14.530641 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.530618    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:14.530776 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.530760    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh" pod "51cb27e2-420d-472f-95cf-2d2ea5359485" (UID: "51cb27e2-420d-472f-95cf-2d2ea5359485"). InnerVolumeSpecName "kube-api-access-bb7xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:14.629550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.629521    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.629550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.629545    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51cb27e2-420d-472f-95cf-2d2ea5359485-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.629550 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.629555    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.629799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.629591    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb7xh\" (UniqueName: \"kubernetes.io/projected/51cb27e2-420d-472f-95cf-2d2ea5359485-kube-api-access-bb7xh\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:14.629799 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:14.629602    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51cb27e2-420d-472f-95cf-2d2ea5359485-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:15.265187 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.265133    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" event={"ID":"10939141-efa0-40c6-8bc8-553f16fe3325","Type":"ContainerStarted","Data":"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"}
Oct 30 19:59:15.265376 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.265356    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:15.267036 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.267005    5085 generic.go:356] "Generic (PLEG): container finished" podID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerID="9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3" exitCode=0
Oct 30 19:59:15.267166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.267065    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerDied","Data":"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"}
Oct 30 19:59:15.267166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.267106    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg" event={"ID":"51cb27e2-420d-472f-95cf-2d2ea5359485","Type":"ContainerDied","Data":"87c3f4de54edc52bbbfd070005c2086c165c7a82c7390dd1a0525086b0a1dfac"}
Oct 30 19:59:15.267166 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.267121    5085 scope.go:122] "RemoveContainer" containerID="9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"
Oct 30 19:59:15.267314 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.267105    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"
Oct 30 19:59:15.280902 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.280878    5085 scope.go:122] "RemoveContainer" containerID="0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"
Oct 30 19:59:15.292140 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.291982    5085 scope.go:122] "RemoveContainer" containerID="9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"
Oct 30 19:59:15.292307 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:15.292287    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3\": container with ID starting with 9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3 not found: ID does not exist" containerID="9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"
Oct 30 19:59:15.292353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.292317    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3"} err="failed to get container status \"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3\": rpc error: code = NotFound desc = could not find container \"9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3\": container with ID starting with 9fa207fed1a85590d85cdea884abca7f4bf7ce666459e7d12bcd0b18acc2f5a3 not found: ID does not exist"
Oct 30 19:59:15.292353 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.292336    5085 scope.go:122] "RemoveContainer" containerID="0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"
Oct 30 19:59:15.292621 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:15.292600    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736\": container with ID starting with 0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736 not found: ID does not exist" containerID="0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"
Oct 30 19:59:15.292693 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.292631    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736"} err="failed to get container status \"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736\": rpc error: code = NotFound desc = could not find container \"0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736\": container with ID starting with 0da0f45904ce055b874459450b3f4b110771b0efc9ffca0ae63054e12c813736 not found: ID does not exist"
Oct 30 19:59:15.383811 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.383757    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" podStartSLOduration=2.514034028 podStartE2EDuration="3.383740792s" totalImagesPullingTime="869.706764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:15.38039742 +0000 UTC m=+2064.884801168" watchObservedRunningTime="2026-10-30 19:59:15.383740792 +0000 UTC m=+2064.888144539"
Oct 30 19:59:15.488820 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.488790    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:59:15.494605 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.494574    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e67696-openshift-default-77f68fc745-fvgqg"]
Oct 30 19:59:15.714666 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:15.714632    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:59:16.632540 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:16.632507    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" path="/var/lib/kubelet/pods/51cb27e2-420d-472f-95cf-2d2ea5359485/volumes"
Oct 30 19:59:17.712003 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.711970    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 19:59:17.712642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712624    5085 cpu_manager.go:414] "containerMap: removing container" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.712642 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712642    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.712741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712662    5085 cpu_manager.go:414] "containerMap: removing container" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.712741 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712667    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.712812 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712807    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.712845 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.712820    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="51cb27e2-420d-472f-95cf-2d2ea5359485" containerName="istio-proxy"
Oct 30 19:59:17.718006 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.717987    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.815853 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.815826    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a299b8-serving-cert\""
Oct 30 19:59:17.819458 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.819435    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 19:59:17.870221 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.870187    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrt4\" (UniqueName: \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.870402 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.870365    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.870497 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.870471    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.870595 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.870516    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.975066 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.974936    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.975401 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.975372    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.975607 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.975552    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.975794 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:17.975768    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-serving-cert: secret "maas-api-e2e-ipp-a-a299b8-serving-cert" not found
Oct 30 19:59:17.975935 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:17.975906    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls podName:4dd8f076-84c1-4d98-82fb-1e116212b064 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:18.475881093 +0000 UTC m=+2067.980284859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls") pod "maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" (UID: "4dd8f076-84c1-4d98-82fb-1e116212b064") : secret "maas-api-e2e-ipp-a-a299b8-serving-cert" not found
Oct 30 19:59:17.976139 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.975780    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrt4\" (UniqueName: \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.988288 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.988221    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:17.990278 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:17.990243    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:18.036219 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:18.036181    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrt4\" (UniqueName: \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:18.489097 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:18.489062    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:18.491526 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:18.491504    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:18.631883 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:18.631848    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:19.007351 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:19.007320    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 19:59:19.008386 ip-10-0-134-200 kubenswrapper[5085]: W1030 19:59:19.008354    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd8f076_84c1_4d98_82fb_1e116212b064.slice/crio-e0557a8c28da020e86369e94a0b4d2661f73525d2a19f4dd4bf6ebaa0f4f698b WatchSource:0}: Error finding container e0557a8c28da020e86369e94a0b4d2661f73525d2a19f4dd4bf6ebaa0f4f698b: Status 404 returned error can't find the container with id e0557a8c28da020e86369e94a0b4d2661f73525d2a19f4dd4bf6ebaa0f4f698b
Oct 30 19:59:19.300477 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:19.300399    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" event={"ID":"4dd8f076-84c1-4d98-82fb-1e116212b064","Type":"ContainerStarted","Data":"e0557a8c28da020e86369e94a0b4d2661f73525d2a19f4dd4bf6ebaa0f4f698b"}
Oct 30 19:59:20.309073 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:20.309033    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" event={"ID":"4dd8f076-84c1-4d98-82fb-1e116212b064","Type":"ContainerStarted","Data":"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"}
Oct 30 19:59:20.309557 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:20.309099    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:20.429366 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:20.429306    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" podStartSLOduration=2.539673316 podStartE2EDuration="3.429287075s" totalImagesPullingTime="889.613759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:20.427355597 +0000 UTC m=+2069.931759343" watchObservedRunningTime="2026-10-30 19:59:20.429287075 +0000 UTC m=+2069.933690826"
Oct 30 19:59:21.281543 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:21.281513    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 19:59:21.940019 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:21.939982    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:59:22.872718 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:22.872677    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:59:22.983117 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:22.983068    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:59:26.322669 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:26.322637    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 19:59:27.873648 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:27.873553    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" podUID="a028d85a-702d-4ecc-ab04-0d4139776c05" containerName="payload-processing" containerID="cri-o://ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6" gracePeriod=25
Oct 30 19:59:27.986394 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:27.986340    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" podUID="202d5a84-a865-47e6-a245-f30d3a91c17d" containerName="payload-pre-processing" containerID="cri-o://38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269" gracePeriod=25
Oct 30 19:59:28.147299 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.147274    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:59:28.226429 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.226395    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\" (UniqueName: \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\") pod \"a028d85a-702d-4ecc-ab04-0d4139776c05\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") "
Oct 30 19:59:28.226606 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.226480    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\") pod \"a028d85a-702d-4ecc-ab04-0d4139776c05\" (UID: \"a028d85a-702d-4ecc-ab04-0d4139776c05\") "
Oct 30 19:59:28.226933 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.226907    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume" pod "a028d85a-702d-4ecc-ab04-0d4139776c05" (UID: "a028d85a-702d-4ecc-ab04-0d4139776c05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:28.227038 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.227022    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a028d85a-702d-4ecc-ab04-0d4139776c05-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:28.228486 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.228460    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz" pod "a028d85a-702d-4ecc-ab04-0d4139776c05" (UID: "a028d85a-702d-4ecc-ab04-0d4139776c05"). InnerVolumeSpecName "kube-api-access-ms5sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:28.248443 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.248421    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:59:28.327436 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.327404    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\") pod \"202d5a84-a865-47e6-a245-f30d3a91c17d\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") "
Oct 30 19:59:28.327652 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.327514    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\" (UniqueName: \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\") pod \"202d5a84-a865-47e6-a245-f30d3a91c17d\" (UID: \"202d5a84-a865-47e6-a245-f30d3a91c17d\") "
Oct 30 19:59:28.327868 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.327843    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume" pod "202d5a84-a865-47e6-a245-f30d3a91c17d" (UID: "202d5a84-a865-47e6-a245-f30d3a91c17d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:28.327976 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.327962    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5sz\" (UniqueName: \"kubernetes.io/projected/a028d85a-702d-4ecc-ab04-0d4139776c05-kube-api-access-ms5sz\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:28.328033 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.327979    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202d5a84-a865-47e6-a245-f30d3a91c17d-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:28.329730 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.329698    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9" pod "202d5a84-a865-47e6-a245-f30d3a91c17d" (UID: "202d5a84-a865-47e6-a245-f30d3a91c17d"). InnerVolumeSpecName "kube-api-access-5w6h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:28.428901 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.428863    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6h9\" (UniqueName: \"kubernetes.io/projected/202d5a84-a865-47e6-a245-f30d3a91c17d-kube-api-access-5w6h9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:28.438909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.438880    5085 generic.go:356] "Generic (PLEG): container finished" podID="202d5a84-a865-47e6-a245-f30d3a91c17d" containerID="38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269" exitCode=0
Oct 30 19:59:28.439080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.438939    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"
Oct 30 19:59:28.439080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.438961    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" event={"ID":"202d5a84-a865-47e6-a245-f30d3a91c17d","Type":"ContainerDied","Data":"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"}
Oct 30 19:59:28.439080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.439039    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54" event={"ID":"202d5a84-a865-47e6-a245-f30d3a91c17d","Type":"ContainerDied","Data":"26cb19e2bb72232101273c279d60a8e0c93684054d1a355a1023015e0e6b24cd"}
Oct 30 19:59:28.439080 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.439069    5085 scope.go:122] "RemoveContainer" containerID="38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"
Oct 30 19:59:28.440796 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.440770    5085 generic.go:356] "Generic (PLEG): container finished" podID="a028d85a-702d-4ecc-ab04-0d4139776c05" containerID="ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6" exitCode=0
Oct 30 19:59:28.440913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.440809    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" event={"ID":"a028d85a-702d-4ecc-ab04-0d4139776c05","Type":"ContainerDied","Data":"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"}
Oct 30 19:59:28.440913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.440821    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"
Oct 30 19:59:28.440913 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.440849    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk" event={"ID":"a028d85a-702d-4ecc-ab04-0d4139776c05","Type":"ContainerDied","Data":"89a4d2283c603a0b36e7251138354af423c9cc38fa54b19fe9a58f5c011f189a"}
Oct 30 19:59:28.450685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.450669    5085 scope.go:122] "RemoveContainer" containerID="38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"
Oct 30 19:59:28.451159 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:28.451135    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269\": container with ID starting with 38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269 not found: ID does not exist" containerID="38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"
Oct 30 19:59:28.451266 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.451169    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269"} err="failed to get container status \"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269\": rpc error: code = NotFound desc = could not find container \"38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269\": container with ID starting with 38782776e0465945a2355f11bee2621d2ee8d116e90b9de6f2053510a9e8d269 not found: ID does not exist"
Oct 30 19:59:28.451266 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.451190    5085 scope.go:122] "RemoveContainer" containerID="ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"
Oct 30 19:59:28.462235 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.462214    5085 scope.go:122] "RemoveContainer" containerID="ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"
Oct 30 19:59:28.462492 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:28.462474    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6\": container with ID starting with ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6 not found: ID does not exist" containerID="ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"
Oct 30 19:59:28.462584 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.462499    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6"} err="failed to get container status \"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6\": rpc error: code = NotFound desc = could not find container \"ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6\": container with ID starting with ced6ca5a81e9143c09dce9b075fbfc3d574beec722c211ab6c7fb35c364796c6 not found: ID does not exist"
Oct 30 19:59:28.598148 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.598116    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:59:28.604909 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.604880    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-111f9c64-759998544b-zkrvk"]
Oct 30 19:59:28.631801 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.631479    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a028d85a-702d-4ecc-ab04-0d4139776c05" path="/var/lib/kubelet/pods/a028d85a-702d-4ecc-ab04-0d4139776c05/volumes"
Oct 30 19:59:28.711346 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.711255    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:59:28.714423 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:28.714400    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-111f9c64-7676b8f96c-bts54"]
Oct 30 19:59:30.632744 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:30.632710    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202d5a84-a865-47e6-a245-f30d3a91c17d" path="/var/lib/kubelet/pods/202d5a84-a865-47e6-a245-f30d3a91c17d/volumes"
Oct 30 19:59:37.822524 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:37.822487    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:59:37.823070 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:37.822694    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" podUID="2759869e-8581-47f7-881f-bf4dace05e07" containerName="maas-api" containerID="cri-o://e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b" gracePeriod=30
Oct 30 19:59:38.100979 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.100950    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:59:38.162222 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.162188    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\") pod \"2759869e-8581-47f7-881f-bf4dace05e07\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") "
Oct 30 19:59:38.162413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.162241    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") pod \"2759869e-8581-47f7-881f-bf4dace05e07\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") "
Oct 30 19:59:38.162413 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.162300    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\") pod \"2759869e-8581-47f7-881f-bf4dace05e07\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") "
Oct 30 19:59:38.162538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.162505    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca" pod "2759869e-8581-47f7-881f-bf4dace05e07" (UID: "2759869e-8581-47f7-881f-bf4dace05e07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:38.162538 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.162513    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\" (UniqueName: \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\") pod \"2759869e-8581-47f7-881f-bf4dace05e07\" (UID: \"2759869e-8581-47f7-881f-bf4dace05e07\") "
Oct 30 19:59:38.163142 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.163115    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2759869e-8581-47f7-881f-bf4dace05e07-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:38.164587 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.164530    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462" pod "2759869e-8581-47f7-881f-bf4dace05e07" (UID: "2759869e-8581-47f7-881f-bf4dace05e07"). InnerVolumeSpecName "kube-api-access-ws462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:38.164685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.164530    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls" pod "2759869e-8581-47f7-881f-bf4dace05e07" (UID: "2759869e-8581-47f7-881f-bf4dace05e07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:38.164685 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.164630    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs" pod "2759869e-8581-47f7-881f-bf4dace05e07" (UID: "2759869e-8581-47f7-881f-bf4dace05e07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:38.265297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.265258    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:38.265297 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.265297    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2759869e-8581-47f7-881f-bf4dace05e07-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:38.265549 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.265313    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws462\" (UniqueName: \"kubernetes.io/projected/2759869e-8581-47f7-881f-bf4dace05e07-kube-api-access-ws462\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:59:38.537993 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.537959    5085 generic.go:356] "Generic (PLEG): container finished" podID="2759869e-8581-47f7-881f-bf4dace05e07" containerID="e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b" exitCode=0
Oct 30 19:59:38.538164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.538003    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"
Oct 30 19:59:38.538164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.538043    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" event={"ID":"2759869e-8581-47f7-881f-bf4dace05e07","Type":"ContainerDied","Data":"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"}
Oct 30 19:59:38.538164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.538098    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc" event={"ID":"2759869e-8581-47f7-881f-bf4dace05e07","Type":"ContainerDied","Data":"b7877d34d69cc7a4965b3491f5035c5c9972af178748b9964510e2169494a9a7"}
Oct 30 19:59:38.538164 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.538124    5085 scope.go:122] "RemoveContainer" containerID="e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"
Oct 30 19:59:38.549360 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.549344    5085 scope.go:122] "RemoveContainer" containerID="e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"
Oct 30 19:59:38.549700 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:38.549680    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b\": container with ID starting with e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b not found: ID does not exist" containerID="e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"
Oct 30 19:59:38.549760 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.549708    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b"} err="failed to get container status \"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b\": rpc error: code = NotFound desc = could not find container \"e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b\": container with ID starting with e37d55744e0790de74c2faef23579e5cf9e7b7bf100a2985680538f8631b599b not found: ID does not exist"
Oct 30 19:59:38.699891 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.699861    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:59:38.703119 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:38.703097    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-111f9c64-6d698d5fc6-6qtzc"]
Oct 30 19:59:40.083850 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:40.083811    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"]
Oct 30 19:59:40.130125 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:40.130092    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-111f9c64-tpgmf"]
Oct 30 19:59:40.631451 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:40.631418    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2759869e-8581-47f7-881f-bf4dace05e07" path="/var/lib/kubelet/pods/2759869e-8581-47f7-881f-bf4dace05e07/volumes"
Oct 30 19:59:40.631894 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:40.631879    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5b8471-7580-494f-be07-a64b2e5dae22" path="/var/lib/kubelet/pods/bc5b8471-7580-494f-be07-a64b2e5dae22/volumes"
Oct 30 19:59:49.881418 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:49.881382    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 19:59:51.187814 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:51.187787    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:59:51.188279 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:51.188021    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 19:59:51.939298 ip-10-0-134-200 kubenswrapper[5085]: E1030 19:59:51.939255    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 19:59:53.187116 ip-10-0-134-200 kubenswrapper[5085]: I1030 19:59:53.187082    5085 scope.go:122] "RemoveContainer" containerID="0d771639cd5a673a5b1ddd464af424a11e7b5d851dc487d70ce7a120d28aa514"
Oct 30 20:00:00.206545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.206360    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:00.208036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208006    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a028d85a-702d-4ecc-ab04-0d4139776c05" containerName="payload-processing"
Oct 30 20:00:00.208036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208034    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a028d85a-702d-4ecc-ab04-0d4139776c05" containerName="payload-processing"
Oct 30 20:00:00.208227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208069    5085 cpu_manager.go:414] "containerMap: removing container" podUID="202d5a84-a865-47e6-a245-f30d3a91c17d" containerName="payload-pre-processing"
Oct 30 20:00:00.208227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208078    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202d5a84-a865-47e6-a245-f30d3a91c17d" containerName="payload-pre-processing"
Oct 30 20:00:00.208227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208092    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2759869e-8581-47f7-881f-bf4dace05e07" containerName="maas-api"
Oct 30 20:00:00.208227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208100    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2759869e-8581-47f7-881f-bf4dace05e07" containerName="maas-api"
Oct 30 20:00:00.208441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208237    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a028d85a-702d-4ecc-ab04-0d4139776c05" containerName="payload-processing"
Oct 30 20:00:00.208441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208254    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="202d5a84-a865-47e6-a245-f30d3a91c17d" containerName="payload-pre-processing"
Oct 30 20:00:00.208441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.208264    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2759869e-8581-47f7-881f-bf4dace05e07" containerName="maas-api"
Oct 30 20:00:00.213700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.213677    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:00.213814 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.213783    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.217624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.217604    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:00.217756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.217739    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.221634 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.221619    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.271175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.271141    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:00.273368 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.273346    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:00.329604 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.328886    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:00:00.333432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.333307    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:00.335382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335350    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.335522 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335422    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjl6d\" (UniqueName: \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.335522 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335497    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6p5\" (UniqueName: \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.335673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335525    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.335673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335654    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.335772 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.335685    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vkf\" (UniqueName: \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.436874    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjl6d\" (UniqueName: \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.436957    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6p5\" (UniqueName: \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.436989    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.437081    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.437114    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vkf\" (UniqueName: \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.437251    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.438072    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.439618    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.445585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.440417    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.496714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.496645    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjl6d\" (UniqueName: \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\") pod \"maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.496714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.496646    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6p5\" (UniqueName: \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\") pod \"maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.497633 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.497610    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vkf\" (UniqueName: \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\") pod \"maas-api-key-cleanup-29889840-cx8ps\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.528365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.528335    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:00.536441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.536407    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:00.542754 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.542726    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:00.843043 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.842972    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:00:00.848279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.848258    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:00.901057 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.901019    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ffe14b-serving-cert\""
Oct 30 20:00:00.906265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.906238    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:00:00.946537 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.946503    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:00.946757 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.946658    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56htg\" (UniqueName: \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:00.946825 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.946798    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:00.946963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.946925    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:00.949441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:00.949413    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:00.950674 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:00.950649    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b1714c_89ec_45ef_83da_b1c0516b099b.slice/crio-6b89935669305963f6c64f72e637c633952adb32152ceec563b2c76bbac414bd WatchSource:0}: Error finding container 6b89935669305963f6c64f72e637c633952adb32152ceec563b2c76bbac414bd: Status 404 returned error can't find the container with id 6b89935669305963f6c64f72e637c633952adb32152ceec563b2c76bbac414bd
Oct 30 20:00:00.999791 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:00.999763    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fa889a_704d_4140_a3fe_fa3b6bbda831.slice/crio-2a2275f49e961d69bd172814ed17ea862ee6aa2fa259244a68b711c06d66f102 WatchSource:0}: Error finding container 2a2275f49e961d69bd172814ed17ea862ee6aa2fa259244a68b711c06d66f102: Status 404 returned error can't find the container with id 2a2275f49e961d69bd172814ed17ea862ee6aa2fa259244a68b711c06d66f102
Oct 30 20:00:01.001084 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:01.000976    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e1ac3c_bc7b_480f_aecf_85e1441e9a07.slice/crio-637c4fe4fce699a24194e44542240e690a159494835be06007c8b98bcca929ed WatchSource:0}: Error finding container 637c4fe4fce699a24194e44542240e690a159494835be06007c8b98bcca929ed: Status 404 returned error can't find the container with id 637c4fe4fce699a24194e44542240e690a159494835be06007c8b98bcca929ed
Oct 30 20:00:01.001084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.001016    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:01.007995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.006622    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:01.048304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.048269    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.048422 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.048375    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.048486 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:01.048426    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-serving-cert: secret "maas-api-e2e-ipp-a-ffe14b-serving-cert" not found
Oct 30 20:00:01.048486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.048451    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56htg\" (UniqueName: \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.048614 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:01.048514    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls podName:1bbf4531-806f-42d3-84c8-f064c4fc5e51 nodeName:}" failed. No retries permitted until 2026-10-30 20:00:01.548487766 +0000 UTC m=+2111.052891503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls") pod "maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" (UID: "1bbf4531-806f-42d3-84c8-f064c4fc5e51") : secret "maas-api-e2e-ipp-a-ffe14b-serving-cert" not found
Oct 30 20:00:01.048679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.048634    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.049268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.049244    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.050779 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.050759    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.159213 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.159139    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56htg\" (UniqueName: \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.554811 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.554771    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.557355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.557323    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.706703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.706659    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerStarted","Data":"e6e695d94bf654555623388200202ef14452fac87cf5ae07d8fc64ad21efa11b"}
Oct 30 20:00:01.706993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.706732    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerStarted","Data":"6b89935669305963f6c64f72e637c633952adb32152ceec563b2c76bbac414bd"}
Oct 30 20:00:01.708522 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.708494    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerStarted","Data":"da8e629b0e978169f3346d564bbee3443005cef62356938bdccf1de740a17364"}
Oct 30 20:00:01.708659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.708540    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerStarted","Data":"2a2275f49e961d69bd172814ed17ea862ee6aa2fa259244a68b711c06d66f102"}
Oct 30 20:00:01.710068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.710046    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerStarted","Data":"e22ef4239635686719ff6c6e2d3ad121d5318e05d16b57af6369cd52dd4359e2"}
Oct 30 20:00:01.710161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.710095    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerStarted","Data":"637c4fe4fce699a24194e44542240e690a159494835be06007c8b98bcca929ed"}
Oct 30 20:00:01.762301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.762216    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:01.818844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.818787    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" podStartSLOduration=1.818773755 podStartE2EDuration="1.818773755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.815853113 +0000 UTC m=+2111.320256862" watchObservedRunningTime="2026-10-30 20:00:01.818773755 +0000 UTC m=+2111.323177501"
Oct 30 20:00:01.935078 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:01.935019    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" podStartSLOduration=1.935003439 podStartE2EDuration="1.935003439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.932631899 +0000 UTC m=+2111.437035648" watchObservedRunningTime="2026-10-30 20:00:01.935003439 +0000 UTC m=+2111.439407187"
Oct 30 20:00:02.049160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.049105    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" podStartSLOduration=2.049087966 podStartE2EDuration="2.049087966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.046102749 +0000 UTC m=+2111.550506495" watchObservedRunningTime="2026-10-30 20:00:02.049087966 +0000 UTC m=+2111.553491712"
Oct 30 20:00:02.227725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.227699    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:00:02.228940 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:02.228896    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbf4531_806f_42d3_84c8_f064c4fc5e51.slice/crio-03aa03092cd6bb6682a962355ba2efe926db3c4b3003a6f1cc5eb954abd8b828 WatchSource:0}: Error finding container 03aa03092cd6bb6682a962355ba2efe926db3c4b3003a6f1cc5eb954abd8b828: Status 404 returned error can't find the container with id 03aa03092cd6bb6682a962355ba2efe926db3c4b3003a6f1cc5eb954abd8b828
Oct 30 20:00:02.719234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.719202    5085 generic.go:356] "Generic (PLEG): container finished" podID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerID="e6e695d94bf654555623388200202ef14452fac87cf5ae07d8fc64ad21efa11b" exitCode=60
Oct 30 20:00:02.719690 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.719272    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerDied","Data":"e6e695d94bf654555623388200202ef14452fac87cf5ae07d8fc64ad21efa11b"}
Oct 30 20:00:02.719690 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.719611    5085 scope.go:122] "RemoveContainer" containerID="e6e695d94bf654555623388200202ef14452fac87cf5ae07d8fc64ad21efa11b"
Oct 30 20:00:02.720792 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.720756    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" event={"ID":"1bbf4531-806f-42d3-84c8-f064c4fc5e51","Type":"ContainerStarted","Data":"03aa03092cd6bb6682a962355ba2efe926db3c4b3003a6f1cc5eb954abd8b828"}
Oct 30 20:00:02.722394 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.722352    5085 generic.go:356] "Generic (PLEG): container finished" podID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerID="da8e629b0e978169f3346d564bbee3443005cef62356938bdccf1de740a17364" exitCode=60
Oct 30 20:00:02.722501 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.722427    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerDied","Data":"da8e629b0e978169f3346d564bbee3443005cef62356938bdccf1de740a17364"}
Oct 30 20:00:02.722668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.722642    5085 scope.go:122] "RemoveContainer" containerID="da8e629b0e978169f3346d564bbee3443005cef62356938bdccf1de740a17364"
Oct 30 20:00:02.724186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.724169    5085 generic.go:356] "Generic (PLEG): container finished" podID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerID="e22ef4239635686719ff6c6e2d3ad121d5318e05d16b57af6369cd52dd4359e2" exitCode=60
Oct 30 20:00:02.724263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.724247    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerDied","Data":"e22ef4239635686719ff6c6e2d3ad121d5318e05d16b57af6369cd52dd4359e2"}
Oct 30 20:00:02.724468 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:02.724456    5085 scope.go:122] "RemoveContainer" containerID="e22ef4239635686719ff6c6e2d3ad121d5318e05d16b57af6369cd52dd4359e2"
Oct 30 20:00:03.734112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.734072    5085 generic.go:356] "Generic (PLEG): container finished" podID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerID="c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23" exitCode=60
Oct 30 20:00:03.734639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.734155    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerDied","Data":"c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23"}
Oct 30 20:00:03.734639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.734212    5085 scope.go:122] "RemoveContainer" containerID="e6e695d94bf654555623388200202ef14452fac87cf5ae07d8fc64ad21efa11b"
Oct 30 20:00:03.734639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.734519    5085 scope.go:122] "RemoveContainer" containerID="c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23"
Oct 30 20:00:03.734926 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:03.734905    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-cx8ps_odh-ai-gateway-infra(10b1714c-89ec-45ef-83da-b1c0516b099b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b"
Oct 30 20:00:03.736023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.735993    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" event={"ID":"1bbf4531-806f-42d3-84c8-f064c4fc5e51","Type":"ContainerStarted","Data":"2a0b6861ca26b314448d9f971f01eecf854eecab33cc1c39259793be43f25a17"}
Oct 30 20:00:03.736126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.736082    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:03.737933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.737914    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerStarted","Data":"5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"}
Oct 30 20:00:03.740109 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.740089    5085 generic.go:356] "Generic (PLEG): container finished" podID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerID="73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb" exitCode=60
Oct 30 20:00:03.740219 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.740151    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerDied","Data":"73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb"}
Oct 30 20:00:03.740531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.740517    5085 scope.go:122] "RemoveContainer" containerID="73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb"
Oct 30 20:00:03.740793 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:03.740771    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l_odh-ai-gateway-infra(11e1ac3c-bc7b-480f-aecf-85e1441e9a07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07"
Oct 30 20:00:03.748676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:03.748617    5085 scope.go:122] "RemoveContainer" containerID="e22ef4239635686719ff6c6e2d3ad121d5318e05d16b57af6369cd52dd4359e2"
Oct 30 20:00:04.176351 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.176286    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" podStartSLOduration=3.213234502 podStartE2EDuration="4.176266179s" totalImagesPullingTime="963.031677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:04.175116606 +0000 UTC m=+2113.679520353" watchObservedRunningTime="2026-10-30 20:00:04.176266179 +0000 UTC m=+2113.680669927"
Oct 30 20:00:04.749506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.749474    5085 scope.go:122] "RemoveContainer" containerID="c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23"
Oct 30 20:00:04.749967 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:04.749811    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-cx8ps_odh-ai-gateway-infra(10b1714c-89ec-45ef-83da-b1c0516b099b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b"
Oct 30 20:00:04.751175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.751146    5085 generic.go:356] "Generic (PLEG): container finished" podID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerID="5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f" exitCode=60
Oct 30 20:00:04.751282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.751218    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerDied","Data":"5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"}
Oct 30 20:00:04.751282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.751270    5085 scope.go:122] "RemoveContainer" containerID="da8e629b0e978169f3346d564bbee3443005cef62356938bdccf1de740a17364"
Oct 30 20:00:04.751521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.751497    5085 scope.go:122] "RemoveContainer" containerID="5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"
Oct 30 20:00:04.751831 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:04.751797    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j_odh-ai-gateway-infra(94fa889a-704d-4140-a3fe-fa3b6bbda831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831"
Oct 30 20:00:04.753249 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:04.753231    5085 scope.go:122] "RemoveContainer" containerID="73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb"
Oct 30 20:00:04.753499 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:04.753479    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l_odh-ai-gateway-infra(11e1ac3c-bc7b-480f-aecf-85e1441e9a07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07"
Oct 30 20:00:05.762482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:05.762450    5085 scope.go:122] "RemoveContainer" containerID="5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"
Oct 30 20:00:05.763082 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:05.762872    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j_odh-ai-gateway-infra(94fa889a-704d-4140-a3fe-fa3b6bbda831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831"
Oct 30 20:00:09.759137 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:09.759108    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:00:14.624385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:14.624350    5085 scope.go:122] "RemoveContainer" containerID="73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb"
Oct 30 20:00:15.624235 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:15.624203    5085 scope.go:122] "RemoveContainer" containerID="5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"
Oct 30 20:00:15.871688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:15.871609    5085 generic.go:356] "Generic (PLEG): container finished" podID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerID="72efd5629716b38b63402c0fea5ef589062236203fb678950af2e482386251c3" exitCode=60
Oct 30 20:00:15.872103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:15.871691    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerDied","Data":"72efd5629716b38b63402c0fea5ef589062236203fb678950af2e482386251c3"}
Oct 30 20:00:15.872103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:15.871761    5085 scope.go:122] "RemoveContainer" containerID="73a49130825cea540d6fef15145df3a5f34f14cf86816b9ecc60303c3f8f1dbb"
Oct 30 20:00:15.872202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:15.872174    5085 scope.go:122] "RemoveContainer" containerID="72efd5629716b38b63402c0fea5ef589062236203fb678950af2e482386251c3"
Oct 30 20:00:15.872525 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:15.872500    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l_odh-ai-gateway-infra(11e1ac3c-bc7b-480f-aecf-85e1441e9a07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07"
Oct 30 20:00:16.623925 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.623897    5085 scope.go:122] "RemoveContainer" containerID="c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23"
Oct 30 20:00:16.882050 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.882020    5085 generic.go:356] "Generic (PLEG): container finished" podID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerID="2ec36a52d414dc8cd043f4b30f057b4f047fa4fc23165fd8f9b8c862d2591f81" exitCode=60
Oct 30 20:00:16.882485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.882094    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerDied","Data":"2ec36a52d414dc8cd043f4b30f057b4f047fa4fc23165fd8f9b8c862d2591f81"}
Oct 30 20:00:16.882485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.882168    5085 scope.go:122] "RemoveContainer" containerID="5ff693cc5f98dd95cc68e7f5a86062cf3c87779a1e46079df14374a78761ad1f"
Oct 30 20:00:16.882622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.882587    5085 scope.go:122] "RemoveContainer" containerID="2ec36a52d414dc8cd043f4b30f057b4f047fa4fc23165fd8f9b8c862d2591f81"
Oct 30 20:00:16.882935 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:16.882911    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j_odh-ai-gateway-infra(94fa889a-704d-4140-a3fe-fa3b6bbda831)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831"
Oct 30 20:00:16.993499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:16.993471    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:17.213457 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.212002    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:17.358693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.358636    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\") pod \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") "
Oct 30 20:00:17.358902 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.358831    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\" (UniqueName: \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\") pod \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\" (UID: \"11e1ac3c-bc7b-480f-aecf-85e1441e9a07\") "
Oct 30 20:00:17.359345 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.359320    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca" pod "11e1ac3c-bc7b-480f-aecf-85e1441e9a07" (UID: "11e1ac3c-bc7b-480f-aecf-85e1441e9a07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:17.359691 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.359662    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:17.361650 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.361619    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5" pod "11e1ac3c-bc7b-480f-aecf-85e1441e9a07" (UID: "11e1ac3c-bc7b-480f-aecf-85e1441e9a07"). InnerVolumeSpecName "kube-api-access-rs6p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:17.465583 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.461769    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6p5\" (UniqueName: \"kubernetes.io/projected/11e1ac3c-bc7b-480f-aecf-85e1441e9a07-kube-api-access-rs6p5\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:17.894530 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.894451    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"
Oct 30 20:00:17.894530 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.894498    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l" event={"ID":"11e1ac3c-bc7b-480f-aecf-85e1441e9a07","Type":"ContainerDied","Data":"637c4fe4fce699a24194e44542240e690a159494835be06007c8b98bcca929ed"}
Oct 30 20:00:17.895051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.894572    5085 scope.go:122] "RemoveContainer" containerID="72efd5629716b38b63402c0fea5ef589062236203fb678950af2e482386251c3"
Oct 30 20:00:17.896723 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.896697    5085 generic.go:356] "Generic (PLEG): container finished" podID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerID="da9940201ec8434720ac0c983ac9019a9b1742b56e8346628d2edbd025c8d6e1" exitCode=60
Oct 30 20:00:17.896845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.896744    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerDied","Data":"da9940201ec8434720ac0c983ac9019a9b1742b56e8346628d2edbd025c8d6e1"}
Oct 30 20:00:17.897088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.897074    5085 scope.go:122] "RemoveContainer" containerID="da9940201ec8434720ac0c983ac9019a9b1742b56e8346628d2edbd025c8d6e1"
Oct 30 20:00:17.897348 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:17.897332    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-cx8ps_odh-ai-gateway-infra(10b1714c-89ec-45ef-83da-b1c0516b099b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b"
Oct 30 20:00:17.907357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:17.907321    5085 scope.go:122] "RemoveContainer" containerID="c4e0e68579ee98e3c8123ddb0e0798a847bfdb3926890e51e58ed1d1976b7b23"
Oct 30 20:00:18.007608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.007555    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:18.167408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.167340    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:18.176071 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.176040    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:18.179841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.179815    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-98a19b-29889840-2nm7l"]
Oct 30 20:00:18.275598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.275541    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\") pod \"94fa889a-704d-4140-a3fe-fa3b6bbda831\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") "
Oct 30 20:00:18.275793 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.275659    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\" (UniqueName: \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\") pod \"94fa889a-704d-4140-a3fe-fa3b6bbda831\" (UID: \"94fa889a-704d-4140-a3fe-fa3b6bbda831\") "
Oct 30 20:00:18.275948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.275926    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca" pod "94fa889a-704d-4140-a3fe-fa3b6bbda831" (UID: "94fa889a-704d-4140-a3fe-fa3b6bbda831"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:18.276168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.276149    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa889a-704d-4140-a3fe-fa3b6bbda831-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.277873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.277848    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d" pod "94fa889a-704d-4140-a3fe-fa3b6bbda831" (UID: "94fa889a-704d-4140-a3fe-fa3b6bbda831"). InnerVolumeSpecName "kube-api-access-bjl6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:18.382584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.381833    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjl6d\" (UniqueName: \"kubernetes.io/projected/94fa889a-704d-4140-a3fe-fa3b6bbda831-kube-api-access-bjl6d\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:18.638552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.638521    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" path="/var/lib/kubelet/pods/11e1ac3c-bc7b-480f-aecf-85e1441e9a07/volumes"
Oct 30 20:00:18.906115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.906030    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"
Oct 30 20:00:18.906115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.906061    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j" event={"ID":"94fa889a-704d-4140-a3fe-fa3b6bbda831","Type":"ContainerDied","Data":"2a2275f49e961d69bd172814ed17ea862ee6aa2fa259244a68b711c06d66f102"}
Oct 30 20:00:18.906614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:18.906120    5085 scope.go:122] "RemoveContainer" containerID="2ec36a52d414dc8cd043f4b30f057b4f047fa4fc23165fd8f9b8c862d2591f81"
Oct 30 20:00:19.066241 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.066033    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:19.070900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.070873    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a74d15-29889840-mv26j"]
Oct 30 20:00:19.073968 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.073943    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:19.252761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.252734    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:19.401377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.401348    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\") pod \"10b1714c-89ec-45ef-83da-b1c0516b099b\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") "
Oct 30 20:00:19.401554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.401408    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\" (UniqueName: \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\") pod \"10b1714c-89ec-45ef-83da-b1c0516b099b\" (UID: \"10b1714c-89ec-45ef-83da-b1c0516b099b\") "
Oct 30 20:00:19.401730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.401706    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca" pod "10b1714c-89ec-45ef-83da-b1c0516b099b" (UID: "10b1714c-89ec-45ef-83da-b1c0516b099b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:19.401947 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.401929    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10b1714c-89ec-45ef-83da-b1c0516b099b-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.403716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.403686    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf" pod "10b1714c-89ec-45ef-83da-b1c0516b099b" (UID: "10b1714c-89ec-45ef-83da-b1c0516b099b"). InnerVolumeSpecName "kube-api-access-w6vkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.503592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.503504    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vkf\" (UniqueName: \"kubernetes.io/projected/10b1714c-89ec-45ef-83da-b1c0516b099b-kube-api-access-w6vkf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.936593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.935280    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"
Oct 30 20:00:19.936593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.935359    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps" event={"ID":"10b1714c-89ec-45ef-83da-b1c0516b099b","Type":"ContainerDied","Data":"6b89935669305963f6c64f72e637c633952adb32152ceec563b2c76bbac414bd"}
Oct 30 20:00:19.936593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:19.935444    5085 scope.go:122] "RemoveContainer" containerID="da9940201ec8434720ac0c983ac9019a9b1742b56e8346628d2edbd025c8d6e1"
Oct 30 20:00:20.096287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:20.096252    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:20.142978 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:20.142949    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-cx8ps"]
Oct 30 20:00:20.632932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:20.632896    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" path="/var/lib/kubelet/pods/10b1714c-89ec-45ef-83da-b1c0516b099b/volumes"
Oct 30 20:00:20.633440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:20.633417    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" path="/var/lib/kubelet/pods/94fa889a-704d-4140-a3fe-fa3b6bbda831/volumes"
Oct 30 20:00:21.939770 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:21.939737    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:00:22.323787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.323702    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 20:00:22.323945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.323892    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" podUID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" containerName="maas-api" containerID="cri-o://bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206" gracePeriod=30
Oct 30 20:00:22.614148 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.614124    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 20:00:22.753058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.753019    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\") pod \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") "
Oct 30 20:00:22.753246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.753177    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\" (UniqueName: \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\") pod \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") "
Oct 30 20:00:22.753371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.753349    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") pod \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") "
Oct 30 20:00:22.753533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.753405    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\") pod \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\" (UID: \"d3295acc-a9f4-4c59-be56-426ac5c7cbcf\") "
Oct 30 20:00:22.754431 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.754401    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca" pod "d3295acc-a9f4-4c59-be56-426ac5c7cbcf" (UID: "d3295acc-a9f4-4c59-be56-426ac5c7cbcf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:22.755607 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.755559    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb" pod "d3295acc-a9f4-4c59-be56-426ac5c7cbcf" (UID: "d3295acc-a9f4-4c59-be56-426ac5c7cbcf"). InnerVolumeSpecName "kube-api-access-rs9wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:22.755709 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.755619    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls" pod "d3295acc-a9f4-4c59-be56-426ac5c7cbcf" (UID: "d3295acc-a9f4-4c59-be56-426ac5c7cbcf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:22.755780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.755712    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs" pod "d3295acc-a9f4-4c59-be56-426ac5c7cbcf" (UID: "d3295acc-a9f4-4c59-be56-426ac5c7cbcf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:22.855386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.855308    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:22.855386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.855339    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9wb\" (UniqueName: \"kubernetes.io/projected/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-kube-api-access-rs9wb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:22.855386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.855352    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:22.855386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.855361    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3295acc-a9f4-4c59-be56-426ac5c7cbcf-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:22.963627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.963597    5085 generic.go:356] "Generic (PLEG): container finished" podID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" containerID="bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206" exitCode=0
Oct 30 20:00:22.964026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.963656    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"
Oct 30 20:00:22.964026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.963685    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" event={"ID":"d3295acc-a9f4-4c59-be56-426ac5c7cbcf","Type":"ContainerDied","Data":"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"}
Oct 30 20:00:22.964026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.963740    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld" event={"ID":"d3295acc-a9f4-4c59-be56-426ac5c7cbcf","Type":"ContainerDied","Data":"10bb8416b53ea476d228ccb03b16a145bb042d6bb7005443db90523af7a672b0"}
Oct 30 20:00:22.964026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.963769    5085 scope.go:122] "RemoveContainer" containerID="bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"
Oct 30 20:00:22.974587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.974543    5085 scope.go:122] "RemoveContainer" containerID="bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"
Oct 30 20:00:22.974827 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:22.974810    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206\": container with ID starting with bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206 not found: ID does not exist" containerID="bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"
Oct 30 20:00:22.974883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:22.974845    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206"} err="failed to get container status \"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206\": rpc error: code = NotFound desc = could not find container \"bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206\": container with ID starting with bcb19ee7268d23532cd62e64681bc1343d6eeceea87ee2c59f81f6f188816206 not found: ID does not exist"
Oct 30 20:00:23.122652 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:23.122571    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 20:00:23.168939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:23.168907    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-585635-68859d554c-vwgld"]
Oct 30 20:00:23.576383 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:23.576353    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:00:24.631688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:24.631653    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" path="/var/lib/kubelet/pods/d3295acc-a9f4-4c59-be56-426ac5c7cbcf/volumes"
Oct 30 20:00:25.124739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:25.124701    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"]
Oct 30 20:00:25.171229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:25.171193    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-585635-wzvzr"]
Oct 30 20:00:25.226478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:25.226445    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 20:00:25.248585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:25.240756    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" containerID="cri-o://9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a" gracePeriod=30
Oct 30 20:00:26.631912 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:26.631878    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e84f5eb-004b-4cbb-994d-b0632b4d4889" path="/var/lib/kubelet/pods/1e84f5eb-004b-4cbb-994d-b0632b4d4889/volumes"
Oct 30 20:00:27.849776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.849732    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850597    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" containerName="maas-api"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850619    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" containerName="maas-api"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850661    5085 cpu_manager.go:414] "containerMap: removing container" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850669    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850678    5085 cpu_manager.go:414] "containerMap: removing container" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850686    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850700    5085 cpu_manager.go:414] "containerMap: removing container" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850706    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850721    5085 cpu_manager.go:414] "containerMap: removing container" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850727    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850739    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850745    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.850758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850764    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850771    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850783    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850791    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850826    5085 cpu_manager.go:414] "containerMap: removing container" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850840    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850975    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.850988    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3295acc-a9f4-4c59-be56-426ac5c7cbcf" containerName="maas-api"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851006    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851015    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851024    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b1714c-89ec-45ef-83da-b1c0516b099b" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851036    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851045    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851054    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fa889a-704d-4140-a3fe-fa3b6bbda831" containerName="cleanup"
Oct 30 20:00:27.851478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.851066    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:27.856733 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.856709    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:27.877386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.877347    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:00:27.919493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.919447    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f9b9b\""
Oct 30 20:00:27.919679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:27.919514    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f9b9b-dockercfg-q49pj\""
Oct 30 20:00:28.048557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.048513    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.048838 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.048815    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvd9\" (UniqueName: \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.150389 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.150302    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvd9\" (UniqueName: \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.150389 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.150359    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.151147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.151126    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.208555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.208515    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvd9\" (UniqueName: \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\") pod \"payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.474363 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.474327    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:28.849123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:28.849093    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:00:28.850881 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:28.850846    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bb3b8b_186f_4cb4_ba76_248516644700.slice/crio-db36ac8557761957e3ddbc52f3eebf27c8f3b0c5dfd9f9c10294bd9fd8ed0c2d WatchSource:0}: Error finding container db36ac8557761957e3ddbc52f3eebf27c8f3b0c5dfd9f9c10294bd9fd8ed0c2d: Status 404 returned error can't find the container with id db36ac8557761957e3ddbc52f3eebf27c8f3b0c5dfd9f9c10294bd9fd8ed0c2d
Oct 30 20:00:29.024469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.024432    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" event={"ID":"19bb3b8b-186f-4cb4-ba76-248516644700","Type":"ContainerStarted","Data":"cb97dd8b236b5e28fb0eedb40179b42131c10011ee2ee62c9e1a797ce0524fbf"}
Oct 30 20:00:29.024696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.024492    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" event={"ID":"19bb3b8b-186f-4cb4-ba76-248516644700","Type":"ContainerStarted","Data":"db36ac8557761957e3ddbc52f3eebf27c8f3b0c5dfd9f9c10294bd9fd8ed0c2d"}
Oct 30 20:00:29.024696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.024519    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:29.139762 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.139660    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" podStartSLOduration=2.139644568 podStartE2EDuration="2.139644568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:29.138363524 +0000 UTC m=+2138.642767252" watchObservedRunningTime="2026-10-30 20:00:29.139644568 +0000 UTC m=+2138.644048349"
Oct 30 20:00:29.153077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.153041    5085 patch_prober.go:29] interesting pod/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:00:29.153238 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:29.153111    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:00:30.527762 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.527691    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 20:00:30.582610 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582578    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582638    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582659    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582697    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582755    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582812    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582856    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582886    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.582929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582914    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\") pod \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\" (UID: \"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390\") "
Oct 30 20:00:30.583194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.582942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:30.583253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583203    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:30.583308 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583274    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:30.583421 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583397    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.583517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583426    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.583517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583440    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.583517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583443    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:30.583517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.583479    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:30.585311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.585283    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:30.585311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.585296    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:30.585609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.585590    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.585704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.585676    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n" pod "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" (UID: "d91e1afb-7f26-45fe-bc8d-0c3b8bc41390"). InnerVolumeSpecName "kube-api-access-w8c6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:30.685176 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685148    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.685176 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685176    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.685384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685188    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.685384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685202    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.685384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685215    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8c6n\" (UniqueName: \"kubernetes.io/projected/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-kube-api-access-w8c6n\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:30.685384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:30.685229    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:31.041139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.041104    5085 generic.go:356] "Generic (PLEG): container finished" podID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerID="9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a" exitCode=0
Oct 30 20:00:31.041317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.041151    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerDied","Data":"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"}
Oct 30 20:00:31.041317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.041161    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"
Oct 30 20:00:31.041317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.041199    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf" event={"ID":"d91e1afb-7f26-45fe-bc8d-0c3b8bc41390","Type":"ContainerDied","Data":"57c2262cbfa4c962d4c2fbcc7c0794d22540c0e32d3ae8d85db0287bf6d034fd"}
Oct 30 20:00:31.041317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.041226    5085 scope.go:122] "RemoveContainer" containerID="9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"
Oct 30 20:00:31.051898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.051868    5085 scope.go:122] "RemoveContainer" containerID="2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"
Oct 30 20:00:31.061903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.061878    5085 scope.go:122] "RemoveContainer" containerID="9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"
Oct 30 20:00:31.062442 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:31.062409    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a\": container with ID starting with 9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a not found: ID does not exist" containerID="9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"
Oct 30 20:00:31.062517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.062473    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a"} err="failed to get container status \"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a\": rpc error: code = NotFound desc = could not find container \"9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a\": container with ID starting with 9cc38c4de9c5fb0d4ccc3fa790b4f76ac244d51afe0036f4ab1ce1222eb2e45a not found: ID does not exist"
Oct 30 20:00:31.062517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.062494    5085 scope.go:122] "RemoveContainer" containerID="2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"
Oct 30 20:00:31.062952 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:31.062927    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8\": container with ID starting with 2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8 not found: ID does not exist" containerID="2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"
Oct 30 20:00:31.063040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.062961    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8"} err="failed to get container status \"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8\": rpc error: code = NotFound desc = could not find container \"2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8\": container with ID starting with 2b135e7df136266d51f0a492ffdde2f274f3ae53ecae6c38d09cc5c07f6938f8 not found: ID does not exist"
Oct 30 20:00:31.200970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.200933    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 20:00:31.247751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:31.247716    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-111f9c64-openshift-default-69ffbd44fc-fnfzf"]
Oct 30 20:00:32.370476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:32.370436    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 20:00:32.370986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:32.370679    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" containerID="cri-o://0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f" gracePeriod=30
Oct 30 20:00:32.631516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:32.631445    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" path="/var/lib/kubelet/pods/d91e1afb-7f26-45fe-bc8d-0c3b8bc41390/volumes"
Oct 30 20:00:33.308117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:33.308082    5085 patch_prober.go:29] interesting pod/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:00:33.308296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:33.308144    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:00:35.802829 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.802792    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:00:35.803361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803347    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.803402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803362    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.803402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803380    5085 cpu_manager.go:414] "containerMap: removing container" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:35.803402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803385    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:35.803496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803484    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.803496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.803493    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="11e1ac3c-bc7b-480f-aecf-85e1441e9a07" containerName="cleanup"
Oct 30 20:00:35.809048 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.809027    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.846128 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.846097    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.846285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.846251    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gbp\" (UniqueName: \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.863072 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.863044    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:00:35.926428 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.910955    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eaa6d2-dockercfg-dckm5\""
Oct 30 20:00:35.935279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.918258    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eaa6d2\""
Oct 30 20:00:35.947348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.947308    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.949612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.947740    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gbp\" (UniqueName: \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.949612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.949262    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:35.965378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.965348    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:00:35.966085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.966054    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.966085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.966076    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.966279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.966206    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91e1afb-7f26-45fe-bc8d-0c3b8bc41390" containerName="istio-proxy"
Oct 30 20:00:35.970164 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:35.970138    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.007064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.007030    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gbp\" (UniqueName: \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\") pod \"payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:36.024088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.024051    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:00:36.122343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.122261    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:36.153085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.153048    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.153227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.153094    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fss2r\" (UniqueName: \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.254946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.254907    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.255141 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.254959    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fss2r\" (UniqueName: \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.255676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.255653    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.314167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.314124    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fss2r\" (UniqueName: \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\") pod \"payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.502623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.501955    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:00:36.503702 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:36.503675    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f70d42e_6f74_4c44_99ea_59452fe4f1dd.slice/crio-ad4269ab5fba3b3b7cbd4ee5b354d4e27c0a7987748c7da3c527ba171f6a36e9 WatchSource:0}: Error finding container ad4269ab5fba3b3b7cbd4ee5b354d4e27c0a7987748c7da3c527ba171f6a36e9: Status 404 returned error can't find the container with id ad4269ab5fba3b3b7cbd4ee5b354d4e27c0a7987748c7da3c527ba171f6a36e9
Oct 30 20:00:36.585606 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.585555    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:36.960928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:36.960352    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:00:36.964275 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:36.964225    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd55222_319d_4a60_9ca0_3445273cfce1.slice/crio-d70a0b72ceefa1ef231ab4bcc5e0cb009e633f169a70b0c0a8ec3b72ad28f28b WatchSource:0}: Error finding container d70a0b72ceefa1ef231ab4bcc5e0cb009e633f169a70b0c0a8ec3b72ad28f28b: Status 404 returned error can't find the container with id d70a0b72ceefa1ef231ab4bcc5e0cb009e633f169a70b0c0a8ec3b72ad28f28b
Oct 30 20:00:37.104615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.104574    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" event={"ID":"1f70d42e-6f74-4c44-99ea-59452fe4f1dd","Type":"ContainerStarted","Data":"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"}
Oct 30 20:00:37.107806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.104654    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:37.107806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.104679    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" event={"ID":"1f70d42e-6f74-4c44-99ea-59452fe4f1dd","Type":"ContainerStarted","Data":"ad4269ab5fba3b3b7cbd4ee5b354d4e27c0a7987748c7da3c527ba171f6a36e9"}
Oct 30 20:00:37.107806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.107491    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" event={"ID":"5dd55222-319d-4a60-9ca0-3445273cfce1","Type":"ContainerStarted","Data":"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"}
Oct 30 20:00:37.107806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.107546    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" event={"ID":"5dd55222-319d-4a60-9ca0-3445273cfce1","Type":"ContainerStarted","Data":"d70a0b72ceefa1ef231ab4bcc5e0cb009e633f169a70b0c0a8ec3b72ad28f28b"}
Oct 30 20:00:37.107806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.107600    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:37.217747 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.217639    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" podStartSLOduration=2.217624133 podStartE2EDuration="2.217624133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.21482931 +0000 UTC m=+2146.719233054" watchObservedRunningTime="2026-10-30 20:00:37.217624133 +0000 UTC m=+2146.722027887"
Oct 30 20:00:37.326529 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.326472    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" podStartSLOduration=2.326457509 podStartE2EDuration="2.326457509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.32497875 +0000 UTC m=+2146.829382500" watchObservedRunningTime="2026-10-30 20:00:37.326457509 +0000 UTC m=+2146.830861256"
Oct 30 20:00:37.654584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.654544    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.773895    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.773957    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.773999    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774055    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774096    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774130    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774250    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:37.774477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774454    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:37.775066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774524    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.775066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774638    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.775066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774713    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\") pod \"a81051a2-66d1-4944-a137-211dced84b5f\" (UID: \"a81051a2-66d1-4944-a137-211dced84b5f\") "
Oct 30 20:00:37.775066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.774828    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:37.775325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775100    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:37.775325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775305    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:37.775325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775317    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.775481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775338    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.775481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775352    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.775481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.775369    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a81051a2-66d1-4944-a137-211dced84b5f-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.776796 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.776766    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:37.777071 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.777045    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:37.777865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.777827    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:37.777975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.777870    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq" pod "a81051a2-66d1-4944-a137-211dced84b5f" (UID: "a81051a2-66d1-4944-a137-211dced84b5f"). InnerVolumeSpecName "kube-api-access-vqmjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:37.876872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.876826    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.876872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.876866    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.876872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.876879    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a81051a2-66d1-4944-a137-211dced84b5f-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.876872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.876890    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmjq\" (UniqueName: \"kubernetes.io/projected/a81051a2-66d1-4944-a137-211dced84b5f-kube-api-access-vqmjq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:37.877161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:37.876899    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a81051a2-66d1-4944-a137-211dced84b5f-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:00:38.116378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.116294    5085 generic.go:356] "Generic (PLEG): container finished" podID="a81051a2-66d1-4944-a137-211dced84b5f" containerID="0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f" exitCode=0
Oct 30 20:00:38.116378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.116359    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"
Oct 30 20:00:38.116378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.116370    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerDied","Data":"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"}
Oct 30 20:00:38.116990 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.116415    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx" event={"ID":"a81051a2-66d1-4944-a137-211dced84b5f","Type":"ContainerDied","Data":"022e6f6c40c0550bfc35138d3155ca1e38de47581f5a8b82b07a65f0c221ec1a"}
Oct 30 20:00:38.116990 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.116437    5085 scope.go:122] "RemoveContainer" containerID="0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"
Oct 30 20:00:38.129170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.129148    5085 scope.go:122] "RemoveContainer" containerID="fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"
Oct 30 20:00:38.139322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.139305    5085 scope.go:122] "RemoveContainer" containerID="0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"
Oct 30 20:00:38.139577 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:38.139543    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f\": container with ID starting with 0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f not found: ID does not exist" containerID="0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"
Oct 30 20:00:38.139630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.139584    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f"} err="failed to get container status \"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f\": rpc error: code = NotFound desc = could not find container \"0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f\": container with ID starting with 0b68e7364d2856ae94269d993aefbf1b3a24a4c3d513d97a5b0daf38df89dc7f not found: ID does not exist"
Oct 30 20:00:38.139630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.139601    5085 scope.go:122] "RemoveContainer" containerID="fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"
Oct 30 20:00:38.139811 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:38.139795    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48\": container with ID starting with fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48 not found: ID does not exist" containerID="fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"
Oct 30 20:00:38.139854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.139817    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48"} err="failed to get container status \"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48\": rpc error: code = NotFound desc = could not find container \"fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48\": container with ID starting with fcfae42492a5b245ea042903aaa497fa40839f8d810bedefcd3305427fd0bd48 not found: ID does not exist"
Oct 30 20:00:38.274428 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.274396    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 20:00:38.321261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.321227    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-585635-openshift-default-5b9fc8b859-29hbx"]
Oct 30 20:00:38.632230 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:38.632196    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a81051a2-66d1-4944-a137-211dced84b5f" path="/var/lib/kubelet/pods/a81051a2-66d1-4944-a137-211dced84b5f/volumes"
Oct 30 20:00:40.035540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:40.035509    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:00:42.944926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:42.944893    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:00:42.945442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:42.945428    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:42.945488 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:42.945442    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:42.945551 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:42.945541    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:42.949322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:42.949300    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.007622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.005620    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d2ca9\""
Oct 30 20:00:43.039696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.039661    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxf6g\" (UniqueName: \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.039869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.039719    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.051700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.051671    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d2ca9-dockercfg-qg6k2\""
Oct 30 20:00:43.055115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.055091    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:00:43.139877 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.139847    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:00:43.140435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.140421    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:43.140490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.140437    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:43.140573 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.140541    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a81051a2-66d1-4944-a137-211dced84b5f" containerName="istio-proxy"
Oct 30 20:00:43.141249 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.141222    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxf6g\" (UniqueName: \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.141354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.141289    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.142037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.142017    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.144049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.144035    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.201155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.201070    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:00:43.201319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.201242    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxf6g\" (UniqueName: \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\") pod \"payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.242500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.242466    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.242748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.242727    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.316528 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.316494    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:43.343758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.343731    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.343899 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.343778    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.344371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.344350    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.403296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.403265    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\") pod \"payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.457516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.457440    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:43.698420 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.698386    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:00:43.699287 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:43.699259    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55feddaa_c793_48d8_b129_f54e35fdf3f4.slice/crio-9fff57ac1c98269b09993cac97d401292f7c013be9fc0ff04b9268811b514d37 WatchSource:0}: Error finding container 9fff57ac1c98269b09993cac97d401292f7c013be9fc0ff04b9268811b514d37: Status 404 returned error can't find the container with id 9fff57ac1c98269b09993cac97d401292f7c013be9fc0ff04b9268811b514d37
Oct 30 20:00:43.830964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:43.830939    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:00:43.832037 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:43.831991    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d0edaf_c61f_463c_b5f7_6b3a8af1bb4e.slice/crio-8b0e211fd0f3376f5f58c70e438c25e637bfd873f48bf741de092fbdba2830bc WatchSource:0}: Error finding container 8b0e211fd0f3376f5f58c70e438c25e637bfd873f48bf741de092fbdba2830bc: Status 404 returned error can't find the container with id 8b0e211fd0f3376f5f58c70e438c25e637bfd873f48bf741de092fbdba2830bc
Oct 30 20:00:44.165082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.164976    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" event={"ID":"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e","Type":"ContainerStarted","Data":"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"}
Oct 30 20:00:44.165544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.165162    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:44.165544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.165196    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" event={"ID":"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e","Type":"ContainerStarted","Data":"8b0e211fd0f3376f5f58c70e438c25e637bfd873f48bf741de092fbdba2830bc"}
Oct 30 20:00:44.166632 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.166606    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" event={"ID":"55feddaa-c793-48d8-b129-f54e35fdf3f4","Type":"ContainerStarted","Data":"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"}
Oct 30 20:00:44.166745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.166646    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" event={"ID":"55feddaa-c793-48d8-b129-f54e35fdf3f4","Type":"ContainerStarted","Data":"9fff57ac1c98269b09993cac97d401292f7c013be9fc0ff04b9268811b514d37"}
Oct 30 20:00:44.166745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.166666    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:44.281631 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.281529    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" podStartSLOduration=1.28151564 podStartE2EDuration="1.28151564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:44.280089053 +0000 UTC m=+2153.784492800" watchObservedRunningTime="2026-10-30 20:00:44.28151564 +0000 UTC m=+2153.785919408"
Oct 30 20:00:44.391409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:44.391343    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" podStartSLOduration=2.391325763 podStartE2EDuration="2.391325763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:44.390799322 +0000 UTC m=+2153.895203073" watchObservedRunningTime="2026-10-30 20:00:44.391325763 +0000 UTC m=+2153.895729510"
Oct 30 20:00:48.119195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:48.119164    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:00:48.119819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:48.119802    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:00:51.939228 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:00:51.939195    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:00:53.246341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.246306    5085 scope.go:122] "RemoveContainer" containerID="18e9b9b9635d5356bf912a0c50b17002797a13ef1cc0e5be0f09a9d4c7649a96"
Oct 30 20:00:53.681100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.681058    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:00:53.687020 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.686999    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.741245 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.741204    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:00:53.766183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.766140    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.766183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.766180    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v585\" (UniqueName: \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.784014 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.783986    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d80c7d\""
Oct 30 20:00:53.784174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.784015    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d80c7d-dockercfg-vknpb\""
Oct 30 20:00:53.866963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.866932    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.867133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.866968    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v585\" (UniqueName: \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.867667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.867648    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:53.926662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:53.926624    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v585\" (UniqueName: \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\") pod \"payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:54.002425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.002343    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:54.379496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.379469    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:00:54.381283 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:54.381250    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8221db_b8cf_4b10_842d_3c474b854d34.slice/crio-e83a6e89d4c4bc26120cf9ba8a016a3e6d7212632d48786d751121b0487937c6 WatchSource:0}: Error finding container e83a6e89d4c4bc26120cf9ba8a016a3e6d7212632d48786d751121b0487937c6: Status 404 returned error can't find the container with id e83a6e89d4c4bc26120cf9ba8a016a3e6d7212632d48786d751121b0487937c6
Oct 30 20:00:54.440593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.440542    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:00:54.446003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.445983    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.500460 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.500426    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:00:54.576158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.576127    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.576365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.576193    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf2wl\" (UniqueName: \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.677697 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.677658    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.677902 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.677727    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf2wl\" (UniqueName: \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.678329 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.678307    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.736484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.736447    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf2wl\" (UniqueName: \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\") pod \"payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:54.759862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:54.759838    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:55.131714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:55.131684    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:00:55.132505 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:00:55.132477    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826df846_4fee_4a34_a606_bc8e7efcaf9b.slice/crio-493ba01a8efc1b3a3e62bbd115fe59d9b783f9fb071630400994a7378eaba5ad WatchSource:0}: Error finding container 493ba01a8efc1b3a3e62bbd115fe59d9b783f9fb071630400994a7378eaba5ad: Status 404 returned error can't find the container with id 493ba01a8efc1b3a3e62bbd115fe59d9b783f9fb071630400994a7378eaba5ad
Oct 30 20:00:55.183444 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:55.183421    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:00:55.183550 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:55.183519    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:00:55.253505 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:55.253467    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" event={"ID":"826df846-4fee-4a34-a606-bc8e7efcaf9b","Type":"ContainerStarted","Data":"493ba01a8efc1b3a3e62bbd115fe59d9b783f9fb071630400994a7378eaba5ad"}
Oct 30 20:00:55.254807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:55.254781    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" event={"ID":"7f8221db-b8cf-4b10-842d-3c474b854d34","Type":"ContainerStarted","Data":"e83a6e89d4c4bc26120cf9ba8a016a3e6d7212632d48786d751121b0487937c6"}
Oct 30 20:00:57.270469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.270433    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" event={"ID":"826df846-4fee-4a34-a606-bc8e7efcaf9b","Type":"ContainerStarted","Data":"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"}
Oct 30 20:00:57.270914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.270673    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:00:57.272197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.272172    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" event={"ID":"7f8221db-b8cf-4b10-842d-3c474b854d34","Type":"ContainerStarted","Data":"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"}
Oct 30 20:00:57.272312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.272227    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:00:57.385595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.385512    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" podStartSLOduration=1.3821695489999999 podStartE2EDuration="3.385498675s" totalImagesPullingTime="2.003329126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:57.382024938 +0000 UTC m=+2166.886428685" watchObservedRunningTime="2026-10-30 20:00:57.385498675 +0000 UTC m=+2166.889902421"
Oct 30 20:00:57.494328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:00:57.494218    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" podStartSLOduration=1.913145133 podStartE2EDuration="4.494202814s" totalImagesPullingTime="2.581057681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:57.492083439 +0000 UTC m=+2166.996487189" watchObservedRunningTime="2026-10-30 20:00:57.494202814 +0000 UTC m=+2166.998606610"
Oct 30 20:01:04.198761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:04.198724    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:01:06.373929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.373895    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:01:06.379414 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.379391    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.433362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.433331    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:01:06.475965 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.475937    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a299b8\""
Oct 30 20:01:06.476152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.475980    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a299b8-dockercfg-4mcfz\""
Oct 30 20:01:06.556506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.556474    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.556736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.556522    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvvw\" (UniqueName: \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.658630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.658535    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvvw\" (UniqueName: \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.658874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.658849    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.659573 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.659545    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.717263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.717232    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvvw\" (UniqueName: \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\") pod \"payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:06.992890 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:06.992846    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:07.100176 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.100136    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:01:07.108556 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.108531    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.160149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.160110    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:01:07.268376 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.268302    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txqcl\" (UniqueName: \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.268511 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.268487    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.372442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.372401    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.372691 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.372536    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txqcl\" (UniqueName: \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.374312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.373745    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.392858 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.381644    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:01:07.431784 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.431758    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txqcl\" (UniqueName: \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\") pod \"payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.724769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.724736    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:07.945492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.945453    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:01:07.950979 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.950955    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:07.983772 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.983700    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:07.983927 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.983866    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:07.983992 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.983927    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhls7\" (UniqueName: \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:07.984079 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:07.984041    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.006704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.006680    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8b0edc-serving-cert\""
Oct 30 20:01:08.017206 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.017171    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:01:08.087795 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.087753    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.088090 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.087894    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.088090 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.087956    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhls7\" (UniqueName: \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.088090 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.088009    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.089642 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.089613    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.089800 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:08.089727    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-serving-cert: secret "maas-api-e2e-ipp-b-8b0edc-serving-cert" not found
Oct 30 20:01:08.089800 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:08.089786    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls podName:d10dbe94-800c-4348-8371-f8a65c9f03da nodeName:}" failed. No retries permitted until 2026-10-30 20:01:08.589765533 +0000 UTC m=+2178.094169260 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls") pod "maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" (UID: "d10dbe94-800c-4348-8371-f8a65c9f03da") : secret "maas-api-e2e-ipp-b-8b0edc-serving-cert" not found
Oct 30 20:01:08.091140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.091116    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.129865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.129840    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:01:08.131426 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:01:08.131395    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6702ce_42cd_43fb_8045_f7ae6d92fbad.slice/crio-1786e9cd3349128452c01e9e18116a1680b58e3d6d8296a1f7c9accab370eb1a WatchSource:0}: Error finding container 1786e9cd3349128452c01e9e18116a1680b58e3d6d8296a1f7c9accab370eb1a: Status 404 returned error can't find the container with id 1786e9cd3349128452c01e9e18116a1680b58e3d6d8296a1f7c9accab370eb1a
Oct 30 20:01:08.151406 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.151379    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhls7\" (UniqueName: \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.283374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.283286    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:01:08.283374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.283372    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:01:08.373473 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.373432    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" event={"ID":"1e6702ce-42cd-43fb-8045-f7ae6d92fbad","Type":"ContainerStarted","Data":"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"}
Oct 30 20:01:08.373676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.373497    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:08.373676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.373521    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" event={"ID":"1e6702ce-42cd-43fb-8045-f7ae6d92fbad","Type":"ContainerStarted","Data":"1786e9cd3349128452c01e9e18116a1680b58e3d6d8296a1f7c9accab370eb1a"}
Oct 30 20:01:08.375200 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.375165    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" event={"ID":"905d9002-54e4-4caf-b594-3ca95d4801ee","Type":"ContainerStarted","Data":"51a7fb322c326163178d61ff3bd597728977a555017b00b8c996897fd90f2593"}
Oct 30 20:01:08.375544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.375224    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" event={"ID":"905d9002-54e4-4caf-b594-3ca95d4801ee","Type":"ContainerStarted","Data":"84692d99f9f5de64ffb4aa1aeb438a7444edd4d34fc1c778022cfe10693a1140"}
Oct 30 20:01:08.375544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.375253    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:08.594880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.594793    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.597341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.597314    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:08.619536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.619478    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" podStartSLOduration=2.619463483 podStartE2EDuration="2.619463483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:08.618661691 +0000 UTC m=+2178.123065437" watchObservedRunningTime="2026-10-30 20:01:08.619463483 +0000 UTC m=+2178.123867230"
Oct 30 20:01:08.729625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.729549    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" podStartSLOduration=1.7295311789999999 podStartE2EDuration="1.729531179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:08.728595015 +0000 UTC m=+2178.232998756" watchObservedRunningTime="2026-10-30 20:01:08.729531179 +0000 UTC m=+2178.233934926"
Oct 30 20:01:08.874544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:08.874437    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:09.277360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:09.275088    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:01:09.383876 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:09.383836    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" event={"ID":"d10dbe94-800c-4348-8371-f8a65c9f03da","Type":"ContainerStarted","Data":"965f5dbe003c2cdbc9e64feceafcaf0af9ce10a24aa5c6a10b56ab09f09ca18d"}
Oct 30 20:01:10.394160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:10.394080    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" event={"ID":"d10dbe94-800c-4348-8371-f8a65c9f03da","Type":"ContainerStarted","Data":"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"}
Oct 30 20:01:10.394160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:10.394149    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:10.509409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:10.509315    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" podStartSLOduration=2.604722425 podStartE2EDuration="3.509302921s" totalImagesPullingTime="904.580496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:10.506198268 +0000 UTC m=+2180.010602014" watchObservedRunningTime="2026-10-30 20:01:10.509302921 +0000 UTC m=+2180.013706667"
Oct 30 20:01:16.407960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:16.407931    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:01:19.387041 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:19.387001    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:01:19.387480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:19.387156    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:01:20.067547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:20.067512    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"]
Oct 30 20:01:20.098881 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:20.098839    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01027a28-jb77h"]
Oct 30 20:01:20.634161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:20.634125    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c1a2d6-9f58-42f2-983d-1e28a8beef2f" path="/var/lib/kubelet/pods/f1c1a2d6-9f58-42f2-983d-1e28a8beef2f/volumes"
Oct 30 20:01:21.939656 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:21.939621    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:01:23.947145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:23.947113    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:01:23.953389 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:23.953367    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.005143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.005107    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3f9b9b-serving-cert\""
Oct 30 20:01:24.019467 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.019436    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.019651 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.019626    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.019793 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.019770    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hm5m\" (UniqueName: \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.019938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.019915    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.055282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.055253    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:01:24.121035 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.121003    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.121220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.121071    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.121220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.121135    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.121220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.121185    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hm5m\" (UniqueName: \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.121399 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:24.121244    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-serving-cert: secret "maas-api-e2e-ipp-b-3f9b9b-serving-cert" not found
Oct 30 20:01:24.121399 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:24.121316    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls podName:2335a48e-e05f-4087-a236-f394358a7913 nodeName:}" failed. No retries permitted until 2026-10-30 20:01:24.62129471 +0000 UTC m=+2194.125698454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls") pod "maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" (UID: "2335a48e-e05f-4087-a236-f394358a7913") : secret "maas-api-e2e-ipp-b-3f9b9b-serving-cert" not found
Oct 30 20:01:24.121794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.121774    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.123535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.123518    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.179727 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.179693    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hm5m\" (UniqueName: \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.628173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.628141    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.630637 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.630613    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:24.868326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:24.868290    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:25.252389 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:25.252358    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:01:25.253204 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:01:25.253175    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2335a48e_e05f_4087_a236_f394358a7913.slice/crio-323aacf8d152b1db857299d7a6215db479507df607570f635c6c9f4330a6d7eb WatchSource:0}: Error finding container 323aacf8d152b1db857299d7a6215db479507df607570f635c6c9f4330a6d7eb: Status 404 returned error can't find the container with id 323aacf8d152b1db857299d7a6215db479507df607570f635c6c9f4330a6d7eb
Oct 30 20:01:25.528396 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:25.528305    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" event={"ID":"2335a48e-e05f-4087-a236-f394358a7913","Type":"ContainerStarted","Data":"323aacf8d152b1db857299d7a6215db479507df607570f635c6c9f4330a6d7eb"}
Oct 30 20:01:26.537474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:26.537425    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" event={"ID":"2335a48e-e05f-4087-a236-f394358a7913","Type":"ContainerStarted","Data":"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"}
Oct 30 20:01:26.537896 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:26.537496    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:26.655879 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:26.655823    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" podStartSLOduration=2.807444334 podStartE2EDuration="3.655808492s" totalImagesPullingTime="848.364158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:26.653476388 +0000 UTC m=+2196.157880136" watchObservedRunningTime="2026-10-30 20:01:26.655808492 +0000 UTC m=+2196.160212604"
Oct 30 20:01:32.551355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:32.551270    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:01:49.839448 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.839414    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:01:49.843681 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.843661    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:49.900520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.900484    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:01:49.909633 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.909607    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2d2ca9-serving-cert\""
Oct 30 20:01:49.953952 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.953918    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:49.954151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.954034    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:49.954151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.954078    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2fv6\" (UniqueName: \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:49.954151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:49.954148    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.055062 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.055028    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.055264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.055093    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.055264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.055123    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2fv6\" (UniqueName: \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.055264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.055163    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.055264 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:50.055254    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-serving-cert: secret "maas-api-e2e-ipp-a-2d2ca9-serving-cert" not found
Oct 30 20:01:50.055494 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:50.055329    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls podName:91dfdce5-e842-448a-8aab-0bfd7291c8e0 nodeName:}" failed. No retries permitted until 2026-10-30 20:01:50.555308618 +0000 UTC m=+2220.059712344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls") pod "maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" (UID: "91dfdce5-e842-448a-8aab-0bfd7291c8e0") : secret "maas-api-e2e-ipp-a-2d2ca9-serving-cert" not found
Oct 30 20:01:50.055734 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.055714    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.057523 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.057504    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.119693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.119624    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2fv6\" (UniqueName: \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.560824 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.560790    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.563199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.563175    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:50.757359 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:50.757327    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:51.132103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:51.132073    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:01:51.133065 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:01:51.133037    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dfdce5_e842_448a_8aab_0bfd7291c8e0.slice/crio-ca9e7297fe448f0518364352741956bb923406963461cf2d19dce4ee432a5562 WatchSource:0}: Error finding container ca9e7297fe448f0518364352741956bb923406963461cf2d19dce4ee432a5562: Status 404 returned error can't find the container with id ca9e7297fe448f0518364352741956bb923406963461cf2d19dce4ee432a5562
Oct 30 20:01:51.720716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:51.720680    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" event={"ID":"91dfdce5-e842-448a-8aab-0bfd7291c8e0","Type":"ContainerStarted","Data":"ca9e7297fe448f0518364352741956bb923406963461cf2d19dce4ee432a5562"}
Oct 30 20:01:51.940627 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:01:51.940592    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:01:52.729653 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:52.729619    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" event={"ID":"91dfdce5-e842-448a-8aab-0bfd7291c8e0","Type":"ContainerStarted","Data":"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"}
Oct 30 20:01:52.730088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:52.729813    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:01:52.795587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:52.786602    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:01:52.851112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:52.850934    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" podStartSLOduration=3.009011375 podStartE2EDuration="3.850912421s" totalImagesPullingTime="841.901046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:52.847612146 +0000 UTC m=+2222.352015893" watchObservedRunningTime="2026-10-30 20:01:52.850912421 +0000 UTC m=+2222.355316170"
Oct 30 20:01:53.313316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:53.313282    5085 scope.go:122] "RemoveContainer" containerID="94073c8733aa680c2f20081b548e754193d71175909c19f51aaf5527bd8574f9"
Oct 30 20:01:53.327320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:53.327293    5085 scope.go:122] "RemoveContainer" containerID="2187b718306551621a872538a36d2567d8b32a9eda2ad989231baeef618be402"
Oct 30 20:01:53.337393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:53.337364    5085 scope.go:122] "RemoveContainer" containerID="8b5326df5f95296ef078fc55fa4358837de9844bd19649d135e8cf1c5b093271"
Oct 30 20:01:58.741841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:01:58.741807    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:02:21.185700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.185664    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:02:21.190385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.190355    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.287438 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.287410    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-77a92c-serving-cert\""
Oct 30 20:02:21.291337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.291307    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:02:21.334797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.334762    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.335166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.334950    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.335166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.334986    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bzc\" (UniqueName: \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.335166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.335036    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.437011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.436910    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.437011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.436964    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bzc\" (UniqueName: \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.437011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.437010    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.437289 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.437115    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.437428 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:02:21.437395    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-serving-cert: secret "maas-api-e2e-ipp-b-77a92c-serving-cert" not found
Oct 30 20:02:21.437512 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:02:21.437501    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls podName:0c7da320-725c-42b5-b6a0-f2d024c8a62c nodeName:}" failed. No retries permitted until 2026-10-30 20:02:21.937479317 +0000 UTC m=+2251.441883044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls") pod "maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" (UID: "0c7da320-725c-42b5-b6a0-f2d024c8a62c") : secret "maas-api-e2e-ipp-b-77a92c-serving-cert" not found
Oct 30 20:02:21.437972 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.437947    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.461226 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.461191    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.496305 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.496264    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bzc\" (UniqueName: \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.943804 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:02:21.943766    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:02:21.947611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.947587    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:21.949899 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:21.949873    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:22.103959 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:22.103928    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:22.478693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:22.478664    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:02:22.480425 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:02:22.480387    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7da320_725c_42b5_b6a0_f2d024c8a62c.slice/crio-71903c7c393c8b0dfb351bd4b2809628b9d11648340c4b3ad243970ac246f507 WatchSource:0}: Error finding container 71903c7c393c8b0dfb351bd4b2809628b9d11648340c4b3ad243970ac246f507: Status 404 returned error can't find the container with id 71903c7c393c8b0dfb351bd4b2809628b9d11648340c4b3ad243970ac246f507
Oct 30 20:02:22.958119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:22.958076    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" event={"ID":"0c7da320-725c-42b5-b6a0-f2d024c8a62c","Type":"ContainerStarted","Data":"71903c7c393c8b0dfb351bd4b2809628b9d11648340c4b3ad243970ac246f507"}
Oct 30 20:02:23.303063 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:23.302967    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:02:23.966264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:23.966228    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" event={"ID":"0c7da320-725c-42b5-b6a0-f2d024c8a62c","Type":"ContainerStarted","Data":"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"}
Oct 30 20:02:23.966692 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:23.966293    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:24.079312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:24.079263    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" podStartSLOduration=2.225402413 podStartE2EDuration="3.079248194s" totalImagesPullingTime="853.845781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:24.07745912 +0000 UTC m=+2253.581862867" watchObservedRunningTime="2026-10-30 20:02:24.079248194 +0000 UTC m=+2253.583651942"
Oct 30 20:02:29.978534 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:29.978505    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:02:51.939739 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:02:51.939696    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:02:51.989862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:51.989827    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:02:53.375152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:53.375120    5085 scope.go:122] "RemoveContainer" containerID="41c271a323c34ebf34daadc400f98e9b9fa014765d0dd329570b27a99de12a7c"
Oct 30 20:02:53.385506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:53.385488    5085 scope.go:122] "RemoveContainer" containerID="e760bac54dd350767a83c49ce84a769137016c51a5f76370f44d452e0f8378b9"
Oct 30 20:02:56.990496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:56.990431    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" podUID="19bb3b8b-186f-4cb4-ba76-248516644700" containerName="payload-processing" containerID="cri-o://cb97dd8b236b5e28fb0eedb40179b42131c10011ee2ee62c9e1a797ce0524fbf" gracePeriod=25
Oct 30 20:02:57.233105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.233063    5085 generic.go:356] "Generic (PLEG): container finished" podID="19bb3b8b-186f-4cb4-ba76-248516644700" containerID="cb97dd8b236b5e28fb0eedb40179b42131c10011ee2ee62c9e1a797ce0524fbf" exitCode=0
Oct 30 20:02:57.233287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.233145    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" event={"ID":"19bb3b8b-186f-4cb4-ba76-248516644700","Type":"ContainerDied","Data":"cb97dd8b236b5e28fb0eedb40179b42131c10011ee2ee62c9e1a797ce0524fbf"}
Oct 30 20:02:57.272831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.272805    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:02:57.313270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.313239    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\" (UniqueName: \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\") pod \"19bb3b8b-186f-4cb4-ba76-248516644700\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") "
Oct 30 20:02:57.313465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.313449    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\") pod \"19bb3b8b-186f-4cb4-ba76-248516644700\" (UID: \"19bb3b8b-186f-4cb4-ba76-248516644700\") "
Oct 30 20:02:57.313825 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.313800    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume" pod "19bb3b8b-186f-4cb4-ba76-248516644700" (UID: "19bb3b8b-186f-4cb4-ba76-248516644700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:57.315353 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.315329    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9" pod "19bb3b8b-186f-4cb4-ba76-248516644700" (UID: "19bb3b8b-186f-4cb4-ba76-248516644700"). InnerVolumeSpecName "kube-api-access-6dvd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:57.415735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.415696    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dvd9\" (UniqueName: \"kubernetes.io/projected/19bb3b8b-186f-4cb4-ba76-248516644700-kube-api-access-6dvd9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:57.415735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:57.415733    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19bb3b8b-186f-4cb4-ba76-248516644700-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:02:58.241994 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.241917    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"
Oct 30 20:02:58.241994 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.241951    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9" event={"ID":"19bb3b8b-186f-4cb4-ba76-248516644700","Type":"ContainerDied","Data":"db36ac8557761957e3ddbc52f3eebf27c8f3b0c5dfd9f9c10294bd9fd8ed0c2d"}
Oct 30 20:02:58.242484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.242014    5085 scope.go:122] "RemoveContainer" containerID="cb97dd8b236b5e28fb0eedb40179b42131c10011ee2ee62c9e1a797ce0524fbf"
Oct 30 20:02:58.400372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.400337    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:02:58.447624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.447592    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f9b9b-7657f5c9b8-pg9z9"]
Oct 30 20:02:58.632098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:02:58.632010    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bb3b8b-186f-4cb4-ba76-248516644700" path="/var/lib/kubelet/pods/19bb3b8b-186f-4cb4-ba76-248516644700/volumes"
Oct 30 20:03:00.052533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:00.052487    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"]
Oct 30 20:03:00.097680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:00.097645    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-226ab710-8h76h"]
Oct 30 20:03:00.428679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:00.428648    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:03:00.632036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:00.632006    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3281494f-8406-4ff8-8b40-ca1fd3ae6979" path="/var/lib/kubelet/pods/3281494f-8406-4ff8-8b40-ca1fd3ae6979/volumes"
Oct 30 20:03:03.641098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:03.641065    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:03:03.758276 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:03.758244    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:03:08.641478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:08.641423    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" podUID="5dd55222-319d-4a60-9ca0-3445273cfce1" containerName="payload-processing" containerID="cri-o://d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301" gracePeriod=25
Oct 30 20:03:08.759028 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:08.758870    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" podUID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" containerName="payload-pre-processing" containerID="cri-o://fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167" gracePeriod=25
Oct 30 20:03:08.949652 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:08.949623    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:03:09.047654 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.047631    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:03:09.075913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.075883    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\") pod \"5dd55222-319d-4a60-9ca0-3445273cfce1\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") "
Oct 30 20:03:09.076084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.075993    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\" (UniqueName: \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\") pod \"5dd55222-319d-4a60-9ca0-3445273cfce1\" (UID: \"5dd55222-319d-4a60-9ca0-3445273cfce1\") "
Oct 30 20:03:09.076302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.076277    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume" pod "5dd55222-319d-4a60-9ca0-3445273cfce1" (UID: "5dd55222-319d-4a60-9ca0-3445273cfce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:09.076637 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.076615    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd55222-319d-4a60-9ca0-3445273cfce1-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:09.078444 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.078421    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r" pod "5dd55222-319d-4a60-9ca0-3445273cfce1" (UID: "5dd55222-319d-4a60-9ca0-3445273cfce1"). InnerVolumeSpecName "kube-api-access-fss2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:09.177175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.177146    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\") pod \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") "
Oct 30 20:03:09.177396 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.177376    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\" (UniqueName: \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\") pod \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\" (UID: \"1f70d42e-6f74-4c44-99ea-59452fe4f1dd\") "
Oct 30 20:03:09.177574 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.177539    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume" pod "1f70d42e-6f74-4c44-99ea-59452fe4f1dd" (UID: "1f70d42e-6f74-4c44-99ea-59452fe4f1dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:09.177848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.177828    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fss2r\" (UniqueName: \"kubernetes.io/projected/5dd55222-319d-4a60-9ca0-3445273cfce1-kube-api-access-fss2r\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:09.177848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.177846    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:09.179466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.179447    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp" pod "1f70d42e-6f74-4c44-99ea-59452fe4f1dd" (UID: "1f70d42e-6f74-4c44-99ea-59452fe4f1dd"). InnerVolumeSpecName "kube-api-access-l9gbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:09.279208 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.279177    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gbp\" (UniqueName: \"kubernetes.io/projected/1f70d42e-6f74-4c44-99ea-59452fe4f1dd-kube-api-access-l9gbp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:09.320546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.320514    5085 generic.go:356] "Generic (PLEG): container finished" podID="5dd55222-319d-4a60-9ca0-3445273cfce1" containerID="d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301" exitCode=0
Oct 30 20:03:09.320702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.320590    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"
Oct 30 20:03:09.320702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.320604    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" event={"ID":"5dd55222-319d-4a60-9ca0-3445273cfce1","Type":"ContainerDied","Data":"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"}
Oct 30 20:03:09.320702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.320667    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp" event={"ID":"5dd55222-319d-4a60-9ca0-3445273cfce1","Type":"ContainerDied","Data":"d70a0b72ceefa1ef231ab4bcc5e0cb009e633f169a70b0c0a8ec3b72ad28f28b"}
Oct 30 20:03:09.320702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.320698    5085 scope.go:122] "RemoveContainer" containerID="d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"
Oct 30 20:03:09.322419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.322281    5085 generic.go:356] "Generic (PLEG): container finished" podID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" containerID="fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167" exitCode=0
Oct 30 20:03:09.322419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.322333    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"
Oct 30 20:03:09.322419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.322334    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" event={"ID":"1f70d42e-6f74-4c44-99ea-59452fe4f1dd","Type":"ContainerDied","Data":"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"}
Oct 30 20:03:09.322419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.322386    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns" event={"ID":"1f70d42e-6f74-4c44-99ea-59452fe4f1dd","Type":"ContainerDied","Data":"ad4269ab5fba3b3b7cbd4ee5b354d4e27c0a7987748c7da3c527ba171f6a36e9"}
Oct 30 20:03:09.333331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.333308    5085 scope.go:122] "RemoveContainer" containerID="d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"
Oct 30 20:03:09.333599 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:09.333575    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301\": container with ID starting with d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301 not found: ID does not exist" containerID="d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"
Oct 30 20:03:09.333674 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.333613    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301"} err="failed to get container status \"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301\": rpc error: code = NotFound desc = could not find container \"d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301\": container with ID starting with d763edb060fc83611e3028ada8159f58181d049acc17b51e2804de64d2916301 not found: ID does not exist"
Oct 30 20:03:09.333674 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.333634    5085 scope.go:122] "RemoveContainer" containerID="fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"
Oct 30 20:03:09.346430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.346402    5085 scope.go:122] "RemoveContainer" containerID="fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"
Oct 30 20:03:09.346779 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:09.346750    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167\": container with ID starting with fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167 not found: ID does not exist" containerID="fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"
Oct 30 20:03:09.346860 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.346787    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167"} err="failed to get container status \"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167\": rpc error: code = NotFound desc = could not find container \"fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167\": container with ID starting with fd1d4fd1b44576c4a9fdb1e921afb388b190fa630dfaf53a2e18928ada32d167 not found: ID does not exist"
Oct 30 20:03:09.478540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.478460    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:03:09.484531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.484501    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eaa6d2-b98b565c4-t4zwp"]
Oct 30 20:03:09.590688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.590655    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:03:09.595012 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:09.594983    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eaa6d2-6779c84bc9-g5wns"]
Oct 30 20:03:10.632107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:10.632075    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" path="/var/lib/kubelet/pods/1f70d42e-6f74-4c44-99ea-59452fe4f1dd/volumes"
Oct 30 20:03:10.632510 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:10.632496    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd55222-319d-4a60-9ca0-3445273cfce1" path="/var/lib/kubelet/pods/5dd55222-319d-4a60-9ca0-3445273cfce1/volumes"
Oct 30 20:03:19.688486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:19.688445    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:03:19.796825 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:19.796793    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:03:21.939444 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:21.939405    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:03:24.689845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.689789    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" podUID="55feddaa-c793-48d8-b129-f54e35fdf3f4" containerName="payload-processing" containerID="cri-o://06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a" gracePeriod=25
Oct 30 20:03:24.797918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.797859    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" podUID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" containerName="payload-pre-processing" containerID="cri-o://3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2" gracePeriod=25
Oct 30 20:03:24.965576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.965539    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:03:24.992296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.992259    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\" (UniqueName: \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\") pod \"55feddaa-c793-48d8-b129-f54e35fdf3f4\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") "
Oct 30 20:03:24.992476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.992319    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\") pod \"55feddaa-c793-48d8-b129-f54e35fdf3f4\" (UID: \"55feddaa-c793-48d8-b129-f54e35fdf3f4\") "
Oct 30 20:03:24.993595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.993549    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume" pod "55feddaa-c793-48d8-b129-f54e35fdf3f4" (UID: "55feddaa-c793-48d8-b129-f54e35fdf3f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:24.996016 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:24.995987    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g" pod "55feddaa-c793-48d8-b129-f54e35fdf3f4" (UID: "55feddaa-c793-48d8-b129-f54e35fdf3f4"). InnerVolumeSpecName "kube-api-access-fxf6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:25.064982 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.064957    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:03:25.093875 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.093841    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\") pod \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") "
Oct 30 20:03:25.094065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.094002    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\") pod \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\" (UID: \"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e\") "
Oct 30 20:03:25.094416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.094390    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume" pod "c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" (UID: "c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:03:25.094626 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.094610    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:25.094708 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.094632    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxf6g\" (UniqueName: \"kubernetes.io/projected/55feddaa-c793-48d8-b129-f54e35fdf3f4-kube-api-access-fxf6g\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:25.094708 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.094657    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55feddaa-c793-48d8-b129-f54e35fdf3f4-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:25.096625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.096603    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj" pod "c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" (UID: "c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e"). InnerVolumeSpecName "kube-api-access-z59tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:03:25.195403 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.195333    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e-kube-api-access-z59tj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:03:25.506883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.506789    5085 generic.go:356] "Generic (PLEG): container finished" podID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" containerID="3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2" exitCode=0
Oct 30 20:03:25.506883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.506844    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"
Oct 30 20:03:25.506883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.506867    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" event={"ID":"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e","Type":"ContainerDied","Data":"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"}
Oct 30 20:03:25.507179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.506924    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz" event={"ID":"c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e","Type":"ContainerDied","Data":"8b0e211fd0f3376f5f58c70e438c25e637bfd873f48bf741de092fbdba2830bc"}
Oct 30 20:03:25.507179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.506947    5085 scope.go:122] "RemoveContainer" containerID="3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"
Oct 30 20:03:25.509451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.509410    5085 generic.go:356] "Generic (PLEG): container finished" podID="55feddaa-c793-48d8-b129-f54e35fdf3f4" containerID="06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a" exitCode=0
Oct 30 20:03:25.509608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.509474    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"
Oct 30 20:03:25.509608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.509447    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" event={"ID":"55feddaa-c793-48d8-b129-f54e35fdf3f4","Type":"ContainerDied","Data":"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"}
Oct 30 20:03:25.509608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.509531    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc" event={"ID":"55feddaa-c793-48d8-b129-f54e35fdf3f4","Type":"ContainerDied","Data":"9fff57ac1c98269b09993cac97d401292f7c013be9fc0ff04b9268811b514d37"}
Oct 30 20:03:25.519540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.519518    5085 scope.go:122] "RemoveContainer" containerID="3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"
Oct 30 20:03:25.519915 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:25.519892    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2\": container with ID starting with 3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2 not found: ID does not exist" containerID="3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"
Oct 30 20:03:25.520023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.519926    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2"} err="failed to get container status \"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2\": rpc error: code = NotFound desc = could not find container \"3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2\": container with ID starting with 3e9e6db7b776192c9debb63bd1a62e0588c5400ae13d73f6a3f9e1e28e993ab2 not found: ID does not exist"
Oct 30 20:03:25.520023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.519949    5085 scope.go:122] "RemoveContainer" containerID="06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"
Oct 30 20:03:25.532024 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.532004    5085 scope.go:122] "RemoveContainer" containerID="06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"
Oct 30 20:03:25.532339 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:25.532317    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a\": container with ID starting with 06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a not found: ID does not exist" containerID="06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"
Oct 30 20:03:25.532406 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.532350    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a"} err="failed to get container status \"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a\": rpc error: code = NotFound desc = could not find container \"06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a\": container with ID starting with 06adbf8635aa6dcbb35d628af28b8051240552d0d7375e9ecbac16abbc6a200a not found: ID does not exist"
Oct 30 20:03:25.665147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.665116    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:03:25.668271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.668243    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2d2ca9-56b6c7589f-vpjcz"]
Oct 30 20:03:25.773503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.773427    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:03:25.781288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:25.781261    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-56b4bf6c58-s7sqc"]
Oct 30 20:03:26.632024 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:26.631987    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55feddaa-c793-48d8-b129-f54e35fdf3f4" path="/var/lib/kubelet/pods/55feddaa-c793-48d8-b129-f54e35fdf3f4/volumes"
Oct 30 20:03:26.632415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:26.632401    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" path="/var/lib/kubelet/pods/c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e/volumes"
Oct 30 20:03:51.940232 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:03:51.940197    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:03:53.096534 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.096502    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:03:53.097134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097117    5085 cpu_manager.go:414] "containerMap: removing container" podUID="19bb3b8b-186f-4cb4-ba76-248516644700" containerName="payload-processing"
Oct 30 20:03:53.097188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097135    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bb3b8b-186f-4cb4-ba76-248516644700" containerName="payload-processing"
Oct 30 20:03:53.097188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097163    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" containerName="payload-pre-processing"
Oct 30 20:03:53.097188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097169    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" containerName="payload-pre-processing"
Oct 30 20:03:53.097188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097176    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5dd55222-319d-4a60-9ca0-3445273cfce1" containerName="payload-processing"
Oct 30 20:03:53.097188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097181    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd55222-319d-4a60-9ca0-3445273cfce1" containerName="payload-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097197    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" containerName="payload-pre-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097203    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" containerName="payload-pre-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097224    5085 cpu_manager.go:414] "containerMap: removing container" podUID="55feddaa-c793-48d8-b129-f54e35fdf3f4" containerName="payload-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097229    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55feddaa-c793-48d8-b129-f54e35fdf3f4" containerName="payload-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097331    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="55feddaa-c793-48d8-b129-f54e35fdf3f4" containerName="payload-processing"
Oct 30 20:03:53.097346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097340    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bb3b8b-186f-4cb4-ba76-248516644700" containerName="payload-processing"
Oct 30 20:03:53.097526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097350    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f70d42e-6f74-4c44-99ea-59452fe4f1dd" containerName="payload-pre-processing"
Oct 30 20:03:53.097526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097357    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd55222-319d-4a60-9ca0-3445273cfce1" containerName="payload-processing"
Oct 30 20:03:53.097526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.097365    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d0edaf-c61f-463c-b5f7-6b3a8af1bb4e" containerName="payload-pre-processing"
Oct 30 20:03:53.101036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.101015    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.155497 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.155459    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8b0edc\""
Oct 30 20:03:53.178378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.178329    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.178544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.178401    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vdb\" (UniqueName: \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.202275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.202240    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8b0edc-dockercfg-7wncr\""
Oct 30 20:03:53.205527 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.205502    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:03:53.280071 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.280037    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.280261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.280082    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vdb\" (UniqueName: \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.280863 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.280837    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.338896 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.338859    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vdb\" (UniqueName: \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\") pod \"payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.415059 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.414981    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:53.416053 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.416033    5085 scope.go:122] "RemoveContainer" containerID="bbdea3f30c1b36793b6fccb14ed99a53f009f55a847afc45371fddf42945c9b0"
Oct 30 20:03:53.428819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.428797    5085 scope.go:122] "RemoveContainer" containerID="e18308e9fde3694738057a492c482b65353f03d5b08e74260bd4e275328a5339"
Oct 30 20:03:53.799692 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:53.799663    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:03:53.806273 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:03:53.805890    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497598bb_c03c_4f3f_91eb_de081b18b984.slice/crio-70f9b17dfb1c5b441471d8879e11a2d3cdf891b06f3b16bf2c97ae1ab369f1a9 WatchSource:0}: Error finding container 70f9b17dfb1c5b441471d8879e11a2d3cdf891b06f3b16bf2c97ae1ab369f1a9: Status 404 returned error can't find the container with id 70f9b17dfb1c5b441471d8879e11a2d3cdf891b06f3b16bf2c97ae1ab369f1a9
Oct 30 20:03:54.750433 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:54.750395    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" event={"ID":"497598bb-c03c-4f3f-91eb-de081b18b984","Type":"ContainerStarted","Data":"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"}
Oct 30 20:03:54.750880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:54.750459    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:03:54.750880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:54.750482    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" event={"ID":"497598bb-c03c-4f3f-91eb-de081b18b984","Type":"ContainerStarted","Data":"70f9b17dfb1c5b441471d8879e11a2d3cdf891b06f3b16bf2c97ae1ab369f1a9"}
Oct 30 20:03:54.863366 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:03:54.863312    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" podStartSLOduration=1.8632994809999999 podStartE2EDuration="1.863299481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:54.859936076 +0000 UTC m=+2344.364339822" watchObservedRunningTime="2026-10-30 20:03:54.863299481 +0000 UTC m=+2344.367703227"
Oct 30 20:04:05.760679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:05.760649    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:04:21.941923 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:21.941888    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:04:47.828222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.828187    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:04:47.832738 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.832719    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:47.930181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.930149    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2d2ca9\""
Oct 30 20:04:47.930354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.930252    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2d2ca9-dockercfg-4jl69\""
Oct 30 20:04:47.935587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.935543    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:04:47.999614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.999578    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:47.999797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:47.999716    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.101119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.101023    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.101119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.101123    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.101784 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.101763    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.160349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.160319    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\") pod \"payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.446621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.446590    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:48.820427 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:48.820400    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:04:48.821337 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:04:48.821311    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c1bf14_656c_4e36_9b68_94c2b23c2997.slice/crio-84655a70b4d5eca6dffe2f9e336f17742ec52015730a05bc71cc101f0663cf08 WatchSource:0}: Error finding container 84655a70b4d5eca6dffe2f9e336f17742ec52015730a05bc71cc101f0663cf08: Status 404 returned error can't find the container with id 84655a70b4d5eca6dffe2f9e336f17742ec52015730a05bc71cc101f0663cf08
Oct 30 20:04:49.144192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:49.144109    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" event={"ID":"73c1bf14-656c-4e36-9b68-94c2b23c2997","Type":"ContainerStarted","Data":"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"}
Oct 30 20:04:49.144192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:49.144160    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" event={"ID":"73c1bf14-656c-4e36-9b68-94c2b23c2997","Type":"ContainerStarted","Data":"84655a70b4d5eca6dffe2f9e336f17742ec52015730a05bc71cc101f0663cf08"}
Oct 30 20:04:49.144664 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:49.144310    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:04:49.256310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:49.256259    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" podStartSLOduration=2.256245753 podStartE2EDuration="2.256245753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:49.255106732 +0000 UTC m=+2398.759510480" watchObservedRunningTime="2026-10-30 20:04:49.256245753 +0000 UTC m=+2398.760649499"
Oct 30 20:04:51.237789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:51.237754    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:04:51.240593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:51.240546    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:04:51.940999 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:51.940959    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:04:53.121642 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.121611    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 20:04:53.122088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.121812    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" podUID="10939141-efa0-40c6-8bc8-553f16fe3325" containerName="maas-api" containerID="cri-o://c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524" gracePeriod=30
Oct 30 20:04:53.418272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.418244    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 20:04:53.461205 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461178    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") pod \"10939141-efa0-40c6-8bc8-553f16fe3325\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") "
Oct 30 20:04:53.461352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461247    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\") pod \"10939141-efa0-40c6-8bc8-553f16fe3325\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") "
Oct 30 20:04:53.461352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461292    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\") pod \"10939141-efa0-40c6-8bc8-553f16fe3325\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") "
Oct 30 20:04:53.461436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461355    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\" (UniqueName: \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\") pod \"10939141-efa0-40c6-8bc8-553f16fe3325\" (UID: \"10939141-efa0-40c6-8bc8-553f16fe3325\") "
Oct 30 20:04:53.461676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461646    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca" pod "10939141-efa0-40c6-8bc8-553f16fe3325" (UID: "10939141-efa0-40c6-8bc8-553f16fe3325"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:53.461807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.461792    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10939141-efa0-40c6-8bc8-553f16fe3325-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.463400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.463375    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs" pod "10939141-efa0-40c6-8bc8-553f16fe3325" (UID: "10939141-efa0-40c6-8bc8-553f16fe3325"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:53.463511 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.463414    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls" pod "10939141-efa0-40c6-8bc8-553f16fe3325" (UID: "10939141-efa0-40c6-8bc8-553f16fe3325"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:53.463511 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.463443    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w" pod "10939141-efa0-40c6-8bc8-553f16fe3325" (UID: "10939141-efa0-40c6-8bc8-553f16fe3325"). InnerVolumeSpecName "kube-api-access-7mw7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:53.576640 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.576602    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.576910 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.576653    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mw7w\" (UniqueName: \"kubernetes.io/projected/10939141-efa0-40c6-8bc8-553f16fe3325-kube-api-access-7mw7w\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.576910 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.576669    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10939141-efa0-40c6-8bc8-553f16fe3325-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.584690 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.584648    5085 scope.go:122] "RemoveContainer" containerID="10f170354159009544035c04078974438d08074f595e4f41745561657f598247"
Oct 30 20:04:53.599123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.599099    5085 scope.go:122] "RemoveContainer" containerID="bdf37272df0facab9b9ded6a309fe6f1477ded0ebf38773d701beed13015241a"
Oct 30 20:04:53.610992 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.610964    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:04:53.611288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.611237    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" podUID="826df846-4fee-4a34-a606-bc8e7efcaf9b" containerName="payload-processing" containerID="cri-o://cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730" gracePeriod=30
Oct 30 20:04:53.665467 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.665383    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:04:53.665653 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.665591    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" podUID="7f8221db-b8cf-4b10-842d-3c474b854d34" containerName="payload-pre-processing" containerID="cri-o://859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe" gracePeriod=30
Oct 30 20:04:53.925858 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.925828    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:04:53.971892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.971858    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:04:53.983074 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983048    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\") pod \"7f8221db-b8cf-4b10-842d-3c474b854d34\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") "
Oct 30 20:04:53.983224 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983096    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\") pod \"826df846-4fee-4a34-a606-bc8e7efcaf9b\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") "
Oct 30 20:04:53.983224 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983131    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\" (UniqueName: \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\") pod \"826df846-4fee-4a34-a606-bc8e7efcaf9b\" (UID: \"826df846-4fee-4a34-a606-bc8e7efcaf9b\") "
Oct 30 20:04:53.983224 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983181    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\" (UniqueName: \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\") pod \"7f8221db-b8cf-4b10-842d-3c474b854d34\" (UID: \"7f8221db-b8cf-4b10-842d-3c474b854d34\") "
Oct 30 20:04:53.983500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983475    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume" pod "7f8221db-b8cf-4b10-842d-3c474b854d34" (UID: "7f8221db-b8cf-4b10-842d-3c474b854d34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:53.983618 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983542    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume" pod "826df846-4fee-4a34-a606-bc8e7efcaf9b" (UID: "826df846-4fee-4a34-a606-bc8e7efcaf9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:04:53.983808 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983790    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826df846-4fee-4a34-a606-bc8e7efcaf9b-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.983860 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.983813    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f8221db-b8cf-4b10-842d-3c474b854d34-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:53.985361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.985335    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl" pod "826df846-4fee-4a34-a606-bc8e7efcaf9b" (UID: "826df846-4fee-4a34-a606-bc8e7efcaf9b"). InnerVolumeSpecName "kube-api-access-rf2wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:53.985439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:53.985378    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585" pod "7f8221db-b8cf-4b10-842d-3c474b854d34" (UID: "7f8221db-b8cf-4b10-842d-3c474b854d34"). InnerVolumeSpecName "kube-api-access-9v585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:54.084491 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.084457    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf2wl\" (UniqueName: \"kubernetes.io/projected/826df846-4fee-4a34-a606-bc8e7efcaf9b-kube-api-access-rf2wl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:54.084491 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.084487    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v585\" (UniqueName: \"kubernetes.io/projected/7f8221db-b8cf-4b10-842d-3c474b854d34-kube-api-access-9v585\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:04:54.211202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.211115    5085 generic.go:356] "Generic (PLEG): container finished" podID="10939141-efa0-40c6-8bc8-553f16fe3325" containerID="c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524" exitCode=0
Oct 30 20:04:54.211591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.211239    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" event={"ID":"10939141-efa0-40c6-8bc8-553f16fe3325","Type":"ContainerDied","Data":"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"}
Oct 30 20:04:54.211591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.211294    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79" event={"ID":"10939141-efa0-40c6-8bc8-553f16fe3325","Type":"ContainerDied","Data":"082840c208499ec69777bc3adaae42d98c1ed4070c50df387255fe3f7d9bcd7a"}
Oct 30 20:04:54.211591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.211326    5085 scope.go:122] "RemoveContainer" containerID="c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"
Oct 30 20:04:54.211591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.211516    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"
Oct 30 20:04:54.214374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.214314    5085 generic.go:356] "Generic (PLEG): container finished" podID="826df846-4fee-4a34-a606-bc8e7efcaf9b" containerID="cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730" exitCode=0
Oct 30 20:04:54.214538 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.214383    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"
Oct 30 20:04:54.214747 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.214718    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" event={"ID":"826df846-4fee-4a34-a606-bc8e7efcaf9b","Type":"ContainerDied","Data":"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"}
Oct 30 20:04:54.214924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.214781    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s" event={"ID":"826df846-4fee-4a34-a606-bc8e7efcaf9b","Type":"ContainerDied","Data":"493ba01a8efc1b3a3e62bbd115fe59d9b783f9fb071630400994a7378eaba5ad"}
Oct 30 20:04:54.216636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.216611    5085 generic.go:356] "Generic (PLEG): container finished" podID="7f8221db-b8cf-4b10-842d-3c474b854d34" containerID="859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe" exitCode=0
Oct 30 20:04:54.216864 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.216848    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" event={"ID":"7f8221db-b8cf-4b10-842d-3c474b854d34","Type":"ContainerDied","Data":"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"}
Oct 30 20:04:54.216940 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.216892    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh" event={"ID":"7f8221db-b8cf-4b10-842d-3c474b854d34","Type":"ContainerDied","Data":"e83a6e89d4c4bc26120cf9ba8a016a3e6d7212632d48786d751121b0487937c6"}
Oct 30 20:04:54.216999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.216987    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"
Oct 30 20:04:54.227750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.227725    5085 scope.go:122] "RemoveContainer" containerID="c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"
Oct 30 20:04:54.228221 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:54.228196    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524\": container with ID starting with c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524 not found: ID does not exist" containerID="c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"
Oct 30 20:04:54.228393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.228366    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524"} err="failed to get container status \"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524\": rpc error: code = NotFound desc = could not find container \"c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524\": container with ID starting with c8f7eae6d8e4f46e75749500a359b3fff57abef6ddbdd264411c485bde769524 not found: ID does not exist"
Oct 30 20:04:54.228491 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.228478    5085 scope.go:122] "RemoveContainer" containerID="cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"
Oct 30 20:04:54.242292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.242273    5085 scope.go:122] "RemoveContainer" containerID="cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"
Oct 30 20:04:54.242625 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:54.242601    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730\": container with ID starting with cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730 not found: ID does not exist" containerID="cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"
Oct 30 20:04:54.242693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.242633    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730"} err="failed to get container status \"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730\": rpc error: code = NotFound desc = could not find container \"cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730\": container with ID starting with cdf1885b1532b717b4d02d9ebd478e9dbf93e924ca0e4701f1f62050eb91e730 not found: ID does not exist"
Oct 30 20:04:54.242693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.242648    5085 scope.go:122] "RemoveContainer" containerID="859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"
Oct 30 20:04:54.252186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.252166    5085 scope.go:122] "RemoveContainer" containerID="859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"
Oct 30 20:04:54.252489 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:54.252470    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe\": container with ID starting with 859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe not found: ID does not exist" containerID="859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"
Oct 30 20:04:54.252554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.252497    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe"} err="failed to get container status \"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe\": rpc error: code = NotFound desc = could not find container \"859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe\": container with ID starting with 859636ad5957a6190bb5a5c602b3afe6987df592e49d3d4797f86da353bc12fe not found: ID does not exist"
Oct 30 20:04:54.370179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.370151    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:04:54.373761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.373721    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d80c7d-7ccb889d7-2rkfh"]
Oct 30 20:04:54.477097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.477016    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:04:54.483134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.483101    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d80c7d-64bd4b8c8-l7h8s"]
Oct 30 20:04:54.585109 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.585074    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 20:04:54.593258 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.593226    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d80c7d-59ddfcfd5-p2j79"]
Oct 30 20:04:54.632176 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.632147    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10939141-efa0-40c6-8bc8-553f16fe3325" path="/var/lib/kubelet/pods/10939141-efa0-40c6-8bc8-553f16fe3325/volumes"
Oct 30 20:04:54.632550 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.632537    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f8221db-b8cf-4b10-842d-3c474b854d34" path="/var/lib/kubelet/pods/7f8221db-b8cf-4b10-842d-3c474b854d34/volumes"
Oct 30 20:04:54.632946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.632932    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826df846-4fee-4a34-a606-bc8e7efcaf9b" path="/var/lib/kubelet/pods/826df846-4fee-4a34-a606-bc8e7efcaf9b/volumes"
Oct 30 20:04:54.831074 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831003    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:04:54.831635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831617    5085 cpu_manager.go:414] "containerMap: removing container" podUID="10939141-efa0-40c6-8bc8-553f16fe3325" containerName="maas-api"
Oct 30 20:04:54.831635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831633    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10939141-efa0-40c6-8bc8-553f16fe3325" containerName="maas-api"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831651    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7f8221db-b8cf-4b10-842d-3c474b854d34" containerName="payload-pre-processing"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831657    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f8221db-b8cf-4b10-842d-3c474b854d34" containerName="payload-pre-processing"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831677    5085 cpu_manager.go:414] "containerMap: removing container" podUID="826df846-4fee-4a34-a606-bc8e7efcaf9b" containerName="payload-processing"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831681    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826df846-4fee-4a34-a606-bc8e7efcaf9b" containerName="payload-processing"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831780    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="10939141-efa0-40c6-8bc8-553f16fe3325" containerName="maas-api"
Oct 30 20:04:54.831788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831788    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f8221db-b8cf-4b10-842d-3c474b854d34" containerName="payload-pre-processing"
Oct 30 20:04:54.832058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.831797    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="826df846-4fee-4a34-a606-bc8e7efcaf9b" containerName="payload-processing"
Oct 30 20:04:54.837125 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.837103    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:54.888543 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.888512    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f700fe-serving-cert\""
Oct 30 20:04:54.892526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.892497    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:04:54.902122 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.902090    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:54.902273 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.902175    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbxx\" (UniqueName: \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:54.902357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.902294    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:54.902357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:54.902331    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.004129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.004093    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.004301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.004248    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.004386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.004373    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.004485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.004470    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbxx\" (UniqueName: \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.004554 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:55.004527    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-serving-cert: secret "maas-api-e2e-ipp-b-f700fe-serving-cert" not found
Oct 30 20:04:55.004626 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:04:55.004614    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls podName:82863459-0ecf-4be4-a316-d1fec173f4f6 nodeName:}" failed. No retries permitted until 2026-10-30 20:04:55.504591646 +0000 UTC m=+2405.008995395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls") pod "maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" (UID: "82863459-0ecf-4be4-a316-d1fec173f4f6") : secret "maas-api-e2e-ipp-b-f700fe-serving-cert" not found
Oct 30 20:04:55.005007 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.004984    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.006622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.006590    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.066418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.066386    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbxx\" (UniqueName: \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.520390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.520357    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.523012 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.522984    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:55.751068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:55.751036    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:56.126287 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:04:56.126253    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82863459_0ecf_4be4_a316_d1fec173f4f6.slice/crio-38836863d759ff11f0601a7f4bef603b122b04c65802e236edab3f23eabcc049 WatchSource:0}: Error finding container 38836863d759ff11f0601a7f4bef603b122b04c65802e236edab3f23eabcc049: Status 404 returned error can't find the container with id 38836863d759ff11f0601a7f4bef603b122b04c65802e236edab3f23eabcc049
Oct 30 20:04:56.127768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:56.127747    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:04:56.154459 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:56.154426    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:04:56.234638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:56.234599    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" event={"ID":"82863459-0ecf-4be4-a316-d1fec173f4f6","Type":"ContainerStarted","Data":"38836863d759ff11f0601a7f4bef603b122b04c65802e236edab3f23eabcc049"}
Oct 30 20:04:56.288588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:56.288390    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:04:57.244216 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:57.244174    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" event={"ID":"82863459-0ecf-4be4-a316-d1fec173f4f6","Type":"ContainerStarted","Data":"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"}
Oct 30 20:04:57.244646 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:57.244416    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:04:57.360321 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:04:57.360219    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" podStartSLOduration=2.419644262 podStartE2EDuration="3.360205972s" totalImagesPullingTime="940.56171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:57.356415853 +0000 UTC m=+2406.860819601" watchObservedRunningTime="2026-10-30 20:04:57.360205972 +0000 UTC m=+2406.864609718"
Oct 30 20:05:00.146178 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.146144    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:05:00.146596 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.146342    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" podUID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" containerName="maas-api" containerID="cri-o://2a0b6861ca26b314448d9f971f01eecf854eecab33cc1c39259793be43f25a17" gracePeriod=30
Oct 30 20:05:00.152987 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.152959    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:05:00.267031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.266992    5085 generic.go:356] "Generic (PLEG): container finished" podID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" containerID="2a0b6861ca26b314448d9f971f01eecf854eecab33cc1c39259793be43f25a17" exitCode=0
Oct 30 20:05:00.267150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.267055    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" event={"ID":"1bbf4531-806f-42d3-84c8-f064c4fc5e51","Type":"ContainerDied","Data":"2a0b6861ca26b314448d9f971f01eecf854eecab33cc1c39259793be43f25a17"}
Oct 30 20:05:00.444914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.444884    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:05:00.590644 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.590611    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\") pod \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") "
Oct 30 20:05:00.590849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.590679    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") pod \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") "
Oct 30 20:05:00.590942 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.590919    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\") pod \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") "
Oct 30 20:05:00.591003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.590961    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\" (UniqueName: \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\") pod \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\" (UID: \"1bbf4531-806f-42d3-84c8-f064c4fc5e51\") "
Oct 30 20:05:00.591075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.591028    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca" pod "1bbf4531-806f-42d3-84c8-f064c4fc5e51" (UID: "1bbf4531-806f-42d3-84c8-f064c4fc5e51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:00.591542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.591511    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf4531-806f-42d3-84c8-f064c4fc5e51-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:00.594116 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.594089    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg" pod "1bbf4531-806f-42d3-84c8-f064c4fc5e51" (UID: "1bbf4531-806f-42d3-84c8-f064c4fc5e51"). InnerVolumeSpecName "kube-api-access-56htg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:00.595744 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.595714    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs" pod "1bbf4531-806f-42d3-84c8-f064c4fc5e51" (UID: "1bbf4531-806f-42d3-84c8-f064c4fc5e51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:00.595885 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.595801    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls" pod "1bbf4531-806f-42d3-84c8-f064c4fc5e51" (UID: "1bbf4531-806f-42d3-84c8-f064c4fc5e51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:00.692499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.692470    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:00.692499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.692499    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56htg\" (UniqueName: \"kubernetes.io/projected/1bbf4531-806f-42d3-84c8-f064c4fc5e51-kube-api-access-56htg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:00.692714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:00.692510    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bbf4531-806f-42d3-84c8-f064c4fc5e51-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:01.277938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.277911    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"
Oct 30 20:05:01.278428 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.277910    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k" event={"ID":"1bbf4531-806f-42d3-84c8-f064c4fc5e51","Type":"ContainerDied","Data":"03aa03092cd6bb6682a962355ba2efe926db3c4b3003a6f1cc5eb954abd8b828"}
Oct 30 20:05:01.278428 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.278047    5085 scope.go:122] "RemoveContainer" containerID="2a0b6861ca26b314448d9f971f01eecf854eecab33cc1c39259793be43f25a17"
Oct 30 20:05:01.436350 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.436313    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:05:01.448868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.448838    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffe14b-7dc9b67f59-whm9k"]
Oct 30 20:05:01.483170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.483139    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 20:05:01.483361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.483341    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" podUID="4dd8f076-84c1-4d98-82fb-1e116212b064" containerName="maas-api" containerID="cri-o://466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1" gracePeriod=30
Oct 30 20:05:01.778582 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.778539    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 20:05:01.912521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.912437    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") pod \"4dd8f076-84c1-4d98-82fb-1e116212b064\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") "
Oct 30 20:05:01.912715 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.912556    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\" (UniqueName: \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\") pod \"4dd8f076-84c1-4d98-82fb-1e116212b064\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") "
Oct 30 20:05:01.912715 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.912700    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\") pod \"4dd8f076-84c1-4d98-82fb-1e116212b064\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") "
Oct 30 20:05:01.912912 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.912737    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\") pod \"4dd8f076-84c1-4d98-82fb-1e116212b064\" (UID: \"4dd8f076-84c1-4d98-82fb-1e116212b064\") "
Oct 30 20:05:01.913187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.913157    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca" pod "4dd8f076-84c1-4d98-82fb-1e116212b064" (UID: "4dd8f076-84c1-4d98-82fb-1e116212b064"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:01.913369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.913275    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd8f076-84c1-4d98-82fb-1e116212b064-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:01.914780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.914751    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs" pod "4dd8f076-84c1-4d98-82fb-1e116212b064" (UID: "4dd8f076-84c1-4d98-82fb-1e116212b064"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:01.914780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.914763    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4" pod "4dd8f076-84c1-4d98-82fb-1e116212b064" (UID: "4dd8f076-84c1-4d98-82fb-1e116212b064"). InnerVolumeSpecName "kube-api-access-zxrt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:01.914977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.914958    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls" pod "4dd8f076-84c1-4d98-82fb-1e116212b064" (UID: "4dd8f076-84c1-4d98-82fb-1e116212b064"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:01.961167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.961127    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:05:01.961365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:01.961339    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" podUID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" containerName="payload-processing" containerID="cri-o://68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6" gracePeriod=30
Oct 30 20:05:02.014623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.014588    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrt4\" (UniqueName: \"kubernetes.io/projected/4dd8f076-84c1-4d98-82fb-1e116212b064-kube-api-access-zxrt4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.014756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.014654    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.014756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.014674    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd8f076-84c1-4d98-82fb-1e116212b064-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.021458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.021431    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:05:02.021682 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.021643    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" podUID="905d9002-54e4-4caf-b594-3ca95d4801ee" containerName="payload-pre-processing" containerID="cri-o://51a7fb322c326163178d61ff3bd597728977a555017b00b8c996897fd90f2593" gracePeriod=30
Oct 30 20:05:02.282686 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.282657    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:05:02.287241 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.287215    5085 generic.go:356] "Generic (PLEG): container finished" podID="4dd8f076-84c1-4d98-82fb-1e116212b064" containerID="466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1" exitCode=0
Oct 30 20:05:02.287399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.287267    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"
Oct 30 20:05:02.287399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.287311    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" event={"ID":"4dd8f076-84c1-4d98-82fb-1e116212b064","Type":"ContainerDied","Data":"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"}
Oct 30 20:05:02.287399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.287381    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr" event={"ID":"4dd8f076-84c1-4d98-82fb-1e116212b064","Type":"ContainerDied","Data":"e0557a8c28da020e86369e94a0b4d2661f73525d2a19f4dd4bf6ebaa0f4f698b"}
Oct 30 20:05:02.287605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.287413    5085 scope.go:122] "RemoveContainer" containerID="466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"
Oct 30 20:05:02.291388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.291368    5085 generic.go:356] "Generic (PLEG): container finished" podID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" containerID="68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6" exitCode=0
Oct 30 20:05:02.291493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.291430    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" event={"ID":"1e6702ce-42cd-43fb-8045-f7ae6d92fbad","Type":"ContainerDied","Data":"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"}
Oct 30 20:05:02.291493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.291459    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"
Oct 30 20:05:02.291493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.291470    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs" event={"ID":"1e6702ce-42cd-43fb-8045-f7ae6d92fbad","Type":"ContainerDied","Data":"1786e9cd3349128452c01e9e18116a1680b58e3d6d8296a1f7c9accab370eb1a"}
Oct 30 20:05:02.293346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.293327    5085 generic.go:356] "Generic (PLEG): container finished" podID="905d9002-54e4-4caf-b594-3ca95d4801ee" containerID="51a7fb322c326163178d61ff3bd597728977a555017b00b8c996897fd90f2593" exitCode=0
Oct 30 20:05:02.293458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.293406    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" event={"ID":"905d9002-54e4-4caf-b594-3ca95d4801ee","Type":"ContainerDied","Data":"51a7fb322c326163178d61ff3bd597728977a555017b00b8c996897fd90f2593"}
Oct 30 20:05:02.311175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.311153    5085 scope.go:122] "RemoveContainer" containerID="466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"
Oct 30 20:05:02.311490 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:02.311473    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1\": container with ID starting with 466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1 not found: ID does not exist" containerID="466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"
Oct 30 20:05:02.311547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.311499    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1"} err="failed to get container status \"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1\": rpc error: code = NotFound desc = could not find container \"466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1\": container with ID starting with 466a8723d949cd1410c525e3f0c91ccc0f2883399eede159787405692ec589d1 not found: ID does not exist"
Oct 30 20:05:02.311547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.311515    5085 scope.go:122] "RemoveContainer" containerID="68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"
Oct 30 20:05:02.323804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.323784    5085 scope.go:122] "RemoveContainer" containerID="68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"
Oct 30 20:05:02.324059 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:02.324036    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6\": container with ID starting with 68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6 not found: ID does not exist" containerID="68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"
Oct 30 20:05:02.324108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.324071    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6"} err="failed to get container status \"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6\": rpc error: code = NotFound desc = could not find container \"68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6\": container with ID starting with 68513e7951b03e4335349e5a75bacee76a055b6be5e1d780626afc8e3f4437e6 not found: ID does not exist"
Oct 30 20:05:02.324814 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.324799    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:05:02.420343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420310    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\" (UniqueName: \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\") pod \"905d9002-54e4-4caf-b594-3ca95d4801ee\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") "
Oct 30 20:05:02.420505 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420481    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\") pod \"905d9002-54e4-4caf-b594-3ca95d4801ee\" (UID: \"905d9002-54e4-4caf-b594-3ca95d4801ee\") "
Oct 30 20:05:02.420599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420510    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\" (UniqueName: \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\") pod \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") "
Oct 30 20:05:02.420599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420554    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\") pod \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\" (UID: \"1e6702ce-42cd-43fb-8045-f7ae6d92fbad\") "
Oct 30 20:05:02.420940 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420912    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume" pod "905d9002-54e4-4caf-b594-3ca95d4801ee" (UID: "905d9002-54e4-4caf-b594-3ca95d4801ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:02.421058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.420984    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume" pod "1e6702ce-42cd-43fb-8045-f7ae6d92fbad" (UID: "1e6702ce-42cd-43fb-8045-f7ae6d92fbad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:02.421615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.421557    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905d9002-54e4-4caf-b594-3ca95d4801ee-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.421689 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.421623    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.422795 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.422771    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl" pod "1e6702ce-42cd-43fb-8045-f7ae6d92fbad" (UID: "1e6702ce-42cd-43fb-8045-f7ae6d92fbad"). InnerVolumeSpecName "kube-api-access-txqcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:02.423009 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.422990    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw" pod "905d9002-54e4-4caf-b594-3ca95d4801ee" (UID: "905d9002-54e4-4caf-b594-3ca95d4801ee"). InnerVolumeSpecName "kube-api-access-zrvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:02.530107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.523081    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvvw\" (UniqueName: \"kubernetes.io/projected/905d9002-54e4-4caf-b594-3ca95d4801ee-kube-api-access-zrvvw\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.530107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.523110    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txqcl\" (UniqueName: \"kubernetes.io/projected/1e6702ce-42cd-43fb-8045-f7ae6d92fbad-kube-api-access-txqcl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.570635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.544616    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 20:05:02.570635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.547342    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a299b8-85fb8fb776-qkjrr"]
Oct 30 20:05:02.633433 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.633401    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" path="/var/lib/kubelet/pods/1bbf4531-806f-42d3-84c8-f064c4fc5e51/volumes"
Oct 30 20:05:02.633878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.633862    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd8f076-84c1-4d98-82fb-1e116212b064" path="/var/lib/kubelet/pods/4dd8f076-84c1-4d98-82fb-1e116212b064/volumes"
Oct 30 20:05:02.784619 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.784511    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:05:02.790021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:02.789993    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a299b8-58bd89c49b-87wrs"]
Oct 30 20:05:03.257662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.257632    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:05:03.303960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.303924    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"
Oct 30 20:05:03.304384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.303959    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr" event={"ID":"905d9002-54e4-4caf-b594-3ca95d4801ee","Type":"ContainerDied","Data":"84692d99f9f5de64ffb4aa1aeb438a7444edd4d34fc1c778022cfe10693a1140"}
Oct 30 20:05:03.304384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.304018    5085 scope.go:122] "RemoveContainer" containerID="51a7fb322c326163178d61ff3bd597728977a555017b00b8c996897fd90f2593"
Oct 30 20:05:03.483701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.483669    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:05:03.487140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:03.487113    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a299b8-7b9767b959-dp9hr"]
Oct 30 20:05:04.632271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:04.632235    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" path="/var/lib/kubelet/pods/1e6702ce-42cd-43fb-8045-f7ae6d92fbad/volumes"
Oct 30 20:05:04.632702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:04.632665    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905d9002-54e4-4caf-b594-3ca95d4801ee" path="/var/lib/kubelet/pods/905d9002-54e4-4caf-b594-3ca95d4801ee/volumes"
Oct 30 20:05:21.939165 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:21.939127    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:05:26.477432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:26.477389    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:05:29.637312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:29.637274    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:05:29.637774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:29.637470    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" podUID="2335a48e-e05f-4087-a236-f394358a7913" containerName="maas-api" containerID="cri-o://00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f" gracePeriod=30
Oct 30 20:05:29.943155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:29.943130    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:05:30.068125 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.068093    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") pod \"2335a48e-e05f-4087-a236-f394358a7913\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") "
Oct 30 20:05:30.068364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.068152    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\") pod \"2335a48e-e05f-4087-a236-f394358a7913\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") "
Oct 30 20:05:30.068364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.068202    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\" (UniqueName: \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\") pod \"2335a48e-e05f-4087-a236-f394358a7913\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") "
Oct 30 20:05:30.068521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.068391    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\") pod \"2335a48e-e05f-4087-a236-f394358a7913\" (UID: \"2335a48e-e05f-4087-a236-f394358a7913\") "
Oct 30 20:05:30.068923 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.068891    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca" pod "2335a48e-e05f-4087-a236-f394358a7913" (UID: "2335a48e-e05f-4087-a236-f394358a7913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:30.070961 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.070918    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs" pod "2335a48e-e05f-4087-a236-f394358a7913" (UID: "2335a48e-e05f-4087-a236-f394358a7913"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:30.070961 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.070941    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m" pod "2335a48e-e05f-4087-a236-f394358a7913" (UID: "2335a48e-e05f-4087-a236-f394358a7913"). InnerVolumeSpecName "kube-api-access-8hm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:30.071614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.071587    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls" pod "2335a48e-e05f-4087-a236-f394358a7913" (UID: "2335a48e-e05f-4087-a236-f394358a7913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:30.170442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.170413    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2335a48e-e05f-4087-a236-f394358a7913-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:30.170442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.170441    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:30.170666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.170452    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2335a48e-e05f-4087-a236-f394358a7913-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:30.170666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.170461    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hm5m\" (UniqueName: \"kubernetes.io/projected/2335a48e-e05f-4087-a236-f394358a7913-kube-api-access-8hm5m\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:30.500689 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.500599    5085 generic.go:356] "Generic (PLEG): container finished" podID="2335a48e-e05f-4087-a236-f394358a7913" containerID="00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f" exitCode=0
Oct 30 20:05:30.500689 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.500642    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"
Oct 30 20:05:30.500932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.500677    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" event={"ID":"2335a48e-e05f-4087-a236-f394358a7913","Type":"ContainerDied","Data":"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"}
Oct 30 20:05:30.500932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.500743    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m" event={"ID":"2335a48e-e05f-4087-a236-f394358a7913","Type":"ContainerDied","Data":"323aacf8d152b1db857299d7a6215db479507df607570f635c6c9f4330a6d7eb"}
Oct 30 20:05:30.500932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.500773    5085 scope.go:122] "RemoveContainer" containerID="00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"
Oct 30 20:05:30.512189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.512171    5085 scope.go:122] "RemoveContainer" containerID="00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"
Oct 30 20:05:30.512489 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:30.512471    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f\": container with ID starting with 00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f not found: ID does not exist" containerID="00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"
Oct 30 20:05:30.512540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.512499    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f"} err="failed to get container status \"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f\": rpc error: code = NotFound desc = could not find container \"00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f\": container with ID starting with 00c1831f8685b00227f8319ac17d2d7dac7299717970c7ea26620c2750acfd2f not found: ID does not exist"
Oct 30 20:05:30.661846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.661812    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:05:30.714081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:30.714050    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3f9b9b-78786f4457-7rt5m"]
Oct 30 20:05:32.634113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:32.634078    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2335a48e-e05f-4087-a236-f394358a7913" path="/var/lib/kubelet/pods/2335a48e-e05f-4087-a236-f394358a7913/volumes"
Oct 30 20:05:48.747451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:48.747414    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 20:05:48.747892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:48.747630    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerName="discovery" containerID="cri-o://1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b" gracePeriod=30
Oct 30 20:05:49.025430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.025397    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 20:05:49.081865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.081836    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.081865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.081874    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.081934    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.081978    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.082175    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.082261    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082363 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.082318    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\") pod \"015a4e2e-3779-458c-b7d4-74c6a149e906\" (UID: \"015a4e2e-3779-458c-b7d4-74c6a149e906\") "
Oct 30 20:05:49.082554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.082509    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:49.082978 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.082937    5085 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-ca-configmap\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.085259 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.085220    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:49.085259 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.085234    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:49.085423 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.085370    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:49.085547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.085524    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:49.085835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.085808    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "kube-api-access-nrpz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:49.086266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.086243    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs" pod "015a4e2e-3779-458c-b7d4-74c6a149e906" (UID: "015a4e2e-3779-458c-b7d4-74c6a149e906"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:49.183848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183811    5085 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-cacerts\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.183848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183843    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.183848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183853    5085 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-kubeconfig\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.184088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183864    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpz8\" (UniqueName: \"kubernetes.io/projected/015a4e2e-3779-458c-b7d4-74c6a149e906-kube-api-access-nrpz8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.184088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183875    5085 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/015a4e2e-3779-458c-b7d4-74c6a149e906-istio-csr-dns-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.184088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.183883    5085 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/015a4e2e-3779-458c-b7d4-74c6a149e906-local-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:05:49.637498 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.637462    5085 generic.go:356] "Generic (PLEG): container finished" podID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerID="1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b" exitCode=0
Oct 30 20:05:49.637687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.637507    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"
Oct 30 20:05:49.637687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.637512    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" event={"ID":"015a4e2e-3779-458c-b7d4-74c6a149e906","Type":"ContainerDied","Data":"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"}
Oct 30 20:05:49.637687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.637634    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z" event={"ID":"015a4e2e-3779-458c-b7d4-74c6a149e906","Type":"ContainerDied","Data":"843a6067f1e248d3cdb4fd55ca6d62988b3f44220b034fd7963cb48821b06ad8"}
Oct 30 20:05:49.637687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.637660    5085 scope.go:122] "RemoveContainer" containerID="1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"
Oct 30 20:05:49.649122 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.649107    5085 scope.go:122] "RemoveContainer" containerID="1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"
Oct 30 20:05:49.649377 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:49.649356    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b\": container with ID starting with 1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b not found: ID does not exist" containerID="1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"
Oct 30 20:05:49.649446 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.649384    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b"} err="failed to get container status \"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b\": rpc error: code = NotFound desc = could not find container \"1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b\": container with ID starting with 1aad48848eb58f869e90571c494dd49063535cf8cf849b661fc87122e31f9f6b not found: ID does not exist"
Oct 30 20:05:49.797068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.797037    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 20:05:49.843941 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:49.843906    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-mj77z"]
Oct 30 20:05:50.634229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:50.634189    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" path="/var/lib/kubelet/pods/015a4e2e-3779-458c-b7d4-74c6a149e906/volumes"
Oct 30 20:05:51.509083 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509044    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:05:51.509616 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509599    5085 cpu_manager.go:414] "containerMap: removing container" podUID="905d9002-54e4-4caf-b594-3ca95d4801ee" containerName="payload-pre-processing"
Oct 30 20:05:51.509616 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509614    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905d9002-54e4-4caf-b594-3ca95d4801ee" containerName="payload-pre-processing"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509632    5085 cpu_manager.go:414] "containerMap: removing container" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerName="discovery"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509638    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerName="discovery"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509651    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" containerName="payload-processing"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509657    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" containerName="payload-processing"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509670    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" containerName="maas-api"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509674    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" containerName="maas-api"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509689    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4dd8f076-84c1-4d98-82fb-1e116212b064" containerName="maas-api"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509693    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd8f076-84c1-4d98-82fb-1e116212b064" containerName="maas-api"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509712    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2335a48e-e05f-4087-a236-f394358a7913" containerName="maas-api"
Oct 30 20:05:51.509750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509717    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2335a48e-e05f-4087-a236-f394358a7913" containerName="maas-api"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509801    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="015a4e2e-3779-458c-b7d4-74c6a149e906" containerName="discovery"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509810    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e6702ce-42cd-43fb-8045-f7ae6d92fbad" containerName="payload-processing"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509818    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bbf4531-806f-42d3-84c8-f064c4fc5e51" containerName="maas-api"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509826    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd8f076-84c1-4d98-82fb-1e116212b064" containerName="maas-api"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509834    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2335a48e-e05f-4087-a236-f394358a7913" containerName="maas-api"
Oct 30 20:05:51.510087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.509840    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="905d9002-54e4-4caf-b594-3ca95d4801ee" containerName="payload-pre-processing"
Oct 30 20:05:51.514481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.514460    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.567834 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.567796    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:05:51.610782 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.610749    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f61bae\""
Oct 30 20:05:51.611095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.610889    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f61bae-dockercfg-x45kj\""
Oct 30 20:05:51.611095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.610976    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxhb\" (UniqueName: \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.611095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.611023    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.672994 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.672959    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:05:51.677046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.677028    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.712026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.711993    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.712193 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.712035    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxhb\" (UniqueName: \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.712193 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.712058    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.712193 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.712177    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2n6l\" (UniqueName: \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.712821 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.712802    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.722011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.721986    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:05:51.772769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.772706    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxhb\" (UniqueName: \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\") pod \"payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.813505 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.813467    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2n6l\" (UniqueName: \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.813715 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.813699    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.814411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.814385    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.830422 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.830400    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:51.873418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.873385    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2n6l\" (UniqueName: \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\") pod \"payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:51.948185 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:05:51.948145    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:05:51.991771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:51.991738    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:52.206183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.206140    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:05:52.206840 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:05:52.206804    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfdcc42_daef_46b2_af7b_66527e8bcaa1.slice/crio-38528221e372f7b788182afc5267abb7e0d28583df7ed238f05e96f3bb130fa6 WatchSource:0}: Error finding container 38528221e372f7b788182afc5267abb7e0d28583df7ed238f05e96f3bb130fa6: Status 404 returned error can't find the container with id 38528221e372f7b788182afc5267abb7e0d28583df7ed238f05e96f3bb130fa6
Oct 30 20:05:52.366651 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.366613    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:05:52.367730 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:05:52.367699    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda091513f_11f1_4cfe_8d1a_7fc257726f91.slice/crio-880a6443b630dd63ffe76dcaa9198c381151a80bd07e1306aedd9ed11571a84b WatchSource:0}: Error finding container 880a6443b630dd63ffe76dcaa9198c381151a80bd07e1306aedd9ed11571a84b: Status 404 returned error can't find the container with id 880a6443b630dd63ffe76dcaa9198c381151a80bd07e1306aedd9ed11571a84b
Oct 30 20:05:52.680738 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.680697    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" event={"ID":"1cfdcc42-daef-46b2-af7b-66527e8bcaa1","Type":"ContainerStarted","Data":"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"}
Oct 30 20:05:52.681171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.680762    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" event={"ID":"1cfdcc42-daef-46b2-af7b-66527e8bcaa1","Type":"ContainerStarted","Data":"38528221e372f7b788182afc5267abb7e0d28583df7ed238f05e96f3bb130fa6"}
Oct 30 20:05:52.681171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.680795    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:05:52.682469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.682446    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" event={"ID":"a091513f-11f1-4cfe-8d1a-7fc257726f91","Type":"ContainerStarted","Data":"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"}
Oct 30 20:05:52.682605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.682482    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" event={"ID":"a091513f-11f1-4cfe-8d1a-7fc257726f91","Type":"ContainerStarted","Data":"880a6443b630dd63ffe76dcaa9198c381151a80bd07e1306aedd9ed11571a84b"}
Oct 30 20:05:52.682605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.682535    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:05:52.793422 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.793369    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" podStartSLOduration=1.793355052 podStartE2EDuration="1.793355052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:52.790777568 +0000 UTC m=+2462.295181315" watchObservedRunningTime="2026-10-30 20:05:52.793355052 +0000 UTC m=+2462.297758798"
Oct 30 20:05:52.904789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:52.904729    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" podStartSLOduration=1.904716096 podStartE2EDuration="1.904716096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:52.902699092 +0000 UTC m=+2462.407102840" watchObservedRunningTime="2026-10-30 20:05:52.904716096 +0000 UTC m=+2462.409119840"
Oct 30 20:05:53.794384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:53.794350    5085 scope.go:122] "RemoveContainer" containerID="7984916013f710e20eae9536e29f021b8942dfc46547124a73375adfcef8d95a"
Oct 30 20:05:58.425725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:58.425649    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 20:05:58.426133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:58.425860    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy" containerID="cri-o://f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91" gracePeriod=30
Oct 30 20:05:58.473153 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:58.473117    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 20:05:58.473359 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:05:58.473313    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy" containerID="cri-o://a058f9502c8d886969fa1af5ddf36459a2a431fe26f0922fbd8e108de0bd9efd" gracePeriod=30
Oct 30 20:06:00.640033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:00.639999    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:06:03.693911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.693823    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:06:03.693911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.693888    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:06:03.735558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.735526    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 20:06:03.755939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.755910    5085 generic.go:356] "Generic (PLEG): container finished" podID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerID="f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91" exitCode=0
Oct 30 20:06:03.756113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.755948    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"
Oct 30 20:06:03.756113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.755984    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" event={"ID":"89cee3d9-2472-45a8-adb9-11a876d6ea21","Type":"ContainerDied","Data":"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"}
Oct 30 20:06:03.756113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.756032    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd" event={"ID":"89cee3d9-2472-45a8-adb9-11a876d6ea21","Type":"ContainerDied","Data":"27a7efc74fdbaa474fa7ed9939bc6f385dc2534fcca5bdb7d1fc58d9515affc8"}
Oct 30 20:06:03.756113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.756062    5085 scope.go:122] "RemoveContainer" containerID="f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"
Oct 30 20:06:03.757851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.757825    5085 generic.go:356] "Generic (PLEG): container finished" podID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerID="a058f9502c8d886969fa1af5ddf36459a2a431fe26f0922fbd8e108de0bd9efd" exitCode=0
Oct 30 20:06:03.757975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.757893    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" event={"ID":"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777","Type":"ContainerDied","Data":"a058f9502c8d886969fa1af5ddf36459a2a431fe26f0922fbd8e108de0bd9efd"}
Oct 30 20:06:03.766162 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.766140    5085 scope.go:122] "RemoveContainer" containerID="f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"
Oct 30 20:06:03.766444 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:03.766425    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91\": container with ID starting with f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91 not found: ID does not exist" containerID="f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"
Oct 30 20:06:03.766513 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.766454    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91"} err="failed to get container status \"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91\": rpc error: code = NotFound desc = could not find container \"f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91\": container with ID starting with f820636df0300f3ed91897a3893abb9c544561b484f38c58ff8c1993cdacee91 not found: ID does not exist"
Oct 30 20:06:03.796319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.796298    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 20:06:03.855871 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.855843    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.855878    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.855927    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.855948    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.855970    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856022    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856051    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856111    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856151    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856173    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856197    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856210    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856238    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856266    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856305    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856347    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856368    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856448    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856515    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856548    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\") pod \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\" (UID: \"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856607    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\") pod \"89cee3d9-2472-45a8-adb9-11a876d6ea21\" (UID: \"89cee3d9-2472-45a8-adb9-11a876d6ea21\") "
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856773    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:03.856868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856800    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.857322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856979    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.857322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.856990    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.857322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857164    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.857322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857278    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.857552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857456    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:03.857628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857552    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857590    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857606    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857620    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857636    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857648    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857662    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857676    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.857841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.857691    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.859075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.859049    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:06:03.859075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.859046    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:06:03.859611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.859553    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "kube-api-access-jbf2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:03.859862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.859839    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:03.859943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.859881    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.860080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.860057    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:03.860080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.860058    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt" pod "89cee3d9-2472-45a8-adb9-11a876d6ea21" (UID: "89cee3d9-2472-45a8-adb9-11a876d6ea21"). InnerVolumeSpecName "kube-api-access-jsbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:03.860446 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.860421    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token" pod "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" (UID: "5e5c97c8-7a5d-4163-ac58-8bcccb4f2777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:03.958861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958781    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbf2c\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-kube-api-access-jbf2c\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.958861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958811    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.958861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958833    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.958861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958842    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsbjt\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-kube-api-access-jsbjt\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.958861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958851    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.959139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958869    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89cee3d9-2472-45a8-adb9-11a876d6ea21-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.959139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958878    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.959139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958886    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89cee3d9-2472-45a8-adb9-11a876d6ea21-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:03.959139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:03.958894    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:04.257168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.257088    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 20:06:04.263875 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.263845    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-fpsfd"]
Oct 30 20:06:04.631188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.631113    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" path="/var/lib/kubelet/pods/89cee3d9-2472-45a8-adb9-11a876d6ea21/volumes"
Oct 30 20:06:04.766312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.766278    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"
Oct 30 20:06:04.766312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.766306    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp" event={"ID":"5e5c97c8-7a5d-4163-ac58-8bcccb4f2777","Type":"ContainerDied","Data":"4b985abc6d09a450314ebdd439226363fdb6b70b520f7bc1f6306b92bcd3e0c9"}
Oct 30 20:06:04.766868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.766366    5085 scope.go:122] "RemoveContainer" containerID="a058f9502c8d886969fa1af5ddf36459a2a431fe26f0922fbd8e108de0bd9efd"
Oct 30 20:06:04.924799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.924763    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 20:06:04.971207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:04.971173    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-694db55c87-p82xp"]
Oct 30 20:06:06.644041 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:06.644007    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" path="/var/lib/kubelet/pods/5e5c97c8-7a5d-4163-ac58-8bcccb4f2777/volumes"
Oct 30 20:06:17.917465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:17.917413    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:06:17.919102 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:17.918868    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" podUID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" containerName="maas-api" containerID="cri-o://c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed" gracePeriod=30
Oct 30 20:06:18.204137 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.204111    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:06:18.321357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.321313    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") pod \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") "
Oct 30 20:06:18.321589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.321382    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\" (UniqueName: \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\") pod \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") "
Oct 30 20:06:18.321589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.321473    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\") pod \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") "
Oct 30 20:06:18.321725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.321624    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\") pod \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\" (UID: \"0c7da320-725c-42b5-b6a0-f2d024c8a62c\") "
Oct 30 20:06:18.326016 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.325942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca" pod "0c7da320-725c-42b5-b6a0-f2d024c8a62c" (UID: "0c7da320-725c-42b5-b6a0-f2d024c8a62c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:18.329623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.326193    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls" pod "0c7da320-725c-42b5-b6a0-f2d024c8a62c" (UID: "0c7da320-725c-42b5-b6a0-f2d024c8a62c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:18.329796 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.329333    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc" pod "0c7da320-725c-42b5-b6a0-f2d024c8a62c" (UID: "0c7da320-725c-42b5-b6a0-f2d024c8a62c"). InnerVolumeSpecName "kube-api-access-67bzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:18.346175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.345996    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs" pod "0c7da320-725c-42b5-b6a0-f2d024c8a62c" (UID: "0c7da320-725c-42b5-b6a0-f2d024c8a62c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:18.423963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.423923    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:18.423963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.423954    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bzc\" (UniqueName: \"kubernetes.io/projected/0c7da320-725c-42b5-b6a0-f2d024c8a62c-kube-api-access-67bzc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:18.423963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.423969    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c7da320-725c-42b5-b6a0-f2d024c8a62c-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:18.424208 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.423981    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7da320-725c-42b5-b6a0-f2d024c8a62c-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:18.855035 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.854995    5085 generic.go:356] "Generic (PLEG): container finished" podID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" containerID="c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed" exitCode=0
Oct 30 20:06:18.855210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.855043    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"
Oct 30 20:06:18.855210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.855083    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" event={"ID":"0c7da320-725c-42b5-b6a0-f2d024c8a62c","Type":"ContainerDied","Data":"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"}
Oct 30 20:06:18.855210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.855138    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58" event={"ID":"0c7da320-725c-42b5-b6a0-f2d024c8a62c","Type":"ContainerDied","Data":"71903c7c393c8b0dfb351bd4b2809628b9d11648340c4b3ad243970ac246f507"}
Oct 30 20:06:18.855210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.855168    5085 scope.go:122] "RemoveContainer" containerID="c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"
Oct 30 20:06:18.865058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.865040    5085 scope.go:122] "RemoveContainer" containerID="c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"
Oct 30 20:06:18.865344 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:18.865326    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed\": container with ID starting with c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed not found: ID does not exist" containerID="c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"
Oct 30 20:06:18.865397 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:18.865356    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed"} err="failed to get container status \"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed\": rpc error: code = NotFound desc = could not find container \"c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed\": container with ID starting with c0d974dde32c488bcd63cc693bb79a0e37eb90cd10c735f62888033f13d068ed not found: ID does not exist"
Oct 30 20:06:19.014926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.014890    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:06:19.020062 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.020028    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-77a92c-5fcfbd84cf-74q58"]
Oct 30 20:06:19.257984 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.257953    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:06:19.258243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.258218    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" podUID="d10dbe94-800c-4348-8371-f8a65c9f03da" containerName="maas-api" containerID="cri-o://a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe" gracePeriod=30
Oct 30 20:06:19.531358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.531295    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:06:19.543030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543003    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\") pod \"d10dbe94-800c-4348-8371-f8a65c9f03da\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") "
Oct 30 20:06:19.543168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543087    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\" (UniqueName: \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\") pod \"d10dbe94-800c-4348-8371-f8a65c9f03da\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") "
Oct 30 20:06:19.543275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543248    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\") pod \"d10dbe94-800c-4348-8371-f8a65c9f03da\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") "
Oct 30 20:06:19.543394 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543379    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") pod \"d10dbe94-800c-4348-8371-f8a65c9f03da\" (UID: \"d10dbe94-800c-4348-8371-f8a65c9f03da\") "
Oct 30 20:06:19.543474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543381    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca" pod "d10dbe94-800c-4348-8371-f8a65c9f03da" (UID: "d10dbe94-800c-4348-8371-f8a65c9f03da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:19.543857 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.543836    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10dbe94-800c-4348-8371-f8a65c9f03da-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:19.545269 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.545246    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7" pod "d10dbe94-800c-4348-8371-f8a65c9f03da" (UID: "d10dbe94-800c-4348-8371-f8a65c9f03da"). InnerVolumeSpecName "kube-api-access-lhls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:19.545269 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.545259    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls" pod "d10dbe94-800c-4348-8371-f8a65c9f03da" (UID: "d10dbe94-800c-4348-8371-f8a65c9f03da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:19.545402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.545283    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs" pod "d10dbe94-800c-4348-8371-f8a65c9f03da" (UID: "d10dbe94-800c-4348-8371-f8a65c9f03da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:19.645003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.644965    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhls7\" (UniqueName: \"kubernetes.io/projected/d10dbe94-800c-4348-8371-f8a65c9f03da-kube-api-access-lhls7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:19.645003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.645002    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:19.645003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.645012    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10dbe94-800c-4348-8371-f8a65c9f03da-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:19.861429 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.861321    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:06:19.861610 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.861581    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" podUID="497598bb-c03c-4f3f-91eb-de081b18b984" containerName="payload-pre-processing" containerID="cri-o://f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea" gracePeriod=30
Oct 30 20:06:19.867270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.867236    5085 generic.go:356] "Generic (PLEG): container finished" podID="d10dbe94-800c-4348-8371-f8a65c9f03da" containerID="a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe" exitCode=0
Oct 30 20:06:19.867425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.867326    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"
Oct 30 20:06:19.867425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.867380    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" event={"ID":"d10dbe94-800c-4348-8371-f8a65c9f03da","Type":"ContainerDied","Data":"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"}
Oct 30 20:06:19.867546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.867437    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk" event={"ID":"d10dbe94-800c-4348-8371-f8a65c9f03da","Type":"ContainerDied","Data":"965f5dbe003c2cdbc9e64feceafcaf0af9ce10a24aa5c6a10b56ab09f09ca18d"}
Oct 30 20:06:19.867546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.867466    5085 scope.go:122] "RemoveContainer" containerID="a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"
Oct 30 20:06:19.919773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.919749    5085 scope.go:122] "RemoveContainer" containerID="a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"
Oct 30 20:06:19.920133 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:19.920104    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe\": container with ID starting with a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe not found: ID does not exist" containerID="a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"
Oct 30 20:06:19.920179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:19.920145    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe"} err="failed to get container status \"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe\": rpc error: code = NotFound desc = could not find container \"a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe\": container with ID starting with a5ce88540cd97c3784302a70afac657a07dc03b7d21e27893c3eb2463c46d5fe not found: ID does not exist"
Oct 30 20:06:20.089492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.088362    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:06:20.093153 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.093126    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b0edc-5c57f8cdb8-5nrbk"]
Oct 30 20:06:20.136210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.136188    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:06:20.151783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.151759    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\") pod \"497598bb-c03c-4f3f-91eb-de081b18b984\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") "
Oct 30 20:06:20.151929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.151838    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\" (UniqueName: \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\") pod \"497598bb-c03c-4f3f-91eb-de081b18b984\" (UID: \"497598bb-c03c-4f3f-91eb-de081b18b984\") "
Oct 30 20:06:20.152149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.152125    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume" pod "497598bb-c03c-4f3f-91eb-de081b18b984" (UID: "497598bb-c03c-4f3f-91eb-de081b18b984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:20.153832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.153810    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb" pod "497598bb-c03c-4f3f-91eb-de081b18b984" (UID: "497598bb-c03c-4f3f-91eb-de081b18b984"). InnerVolumeSpecName "kube-api-access-j4vdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:20.257675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.257635    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/497598bb-c03c-4f3f-91eb-de081b18b984-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:20.257944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.257925    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vdb\" (UniqueName: \"kubernetes.io/projected/497598bb-c03c-4f3f-91eb-de081b18b984-kube-api-access-j4vdb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:20.631590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.631534    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" path="/var/lib/kubelet/pods/0c7da320-725c-42b5-b6a0-f2d024c8a62c/volumes"
Oct 30 20:06:20.632013 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.631998    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10dbe94-800c-4348-8371-f8a65c9f03da" path="/var/lib/kubelet/pods/d10dbe94-800c-4348-8371-f8a65c9f03da/volumes"
Oct 30 20:06:20.875732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.875697    5085 generic.go:356] "Generic (PLEG): container finished" podID="497598bb-c03c-4f3f-91eb-de081b18b984" containerID="f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea" exitCode=0
Oct 30 20:06:20.875897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.875755    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"
Oct 30 20:06:20.875897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.875783    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" event={"ID":"497598bb-c03c-4f3f-91eb-de081b18b984","Type":"ContainerDied","Data":"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"}
Oct 30 20:06:20.875897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.875835    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs" event={"ID":"497598bb-c03c-4f3f-91eb-de081b18b984","Type":"ContainerDied","Data":"70f9b17dfb1c5b441471d8879e11a2d3cdf891b06f3b16bf2c97ae1ab369f1a9"}
Oct 30 20:06:20.875897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.875860    5085 scope.go:122] "RemoveContainer" containerID="f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"
Oct 30 20:06:20.885198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.885183    5085 scope.go:122] "RemoveContainer" containerID="f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"
Oct 30 20:06:20.885468 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:20.885450    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea\": container with ID starting with f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea not found: ID does not exist" containerID="f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"
Oct 30 20:06:20.885520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:20.885477    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea"} err="failed to get container status \"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea\": rpc error: code = NotFound desc = could not find container \"f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea\": container with ID starting with f0e0e0f8c9218e189499951870a9938545a6bb7e4b3a734329546af0661e1aea not found: ID does not exist"
Oct 30 20:06:21.032745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:21.032705    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:06:21.041849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:21.041821    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b0edc-6db5488f59-8fjxs"]
Oct 30 20:06:21.939115 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:21.939067    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:06:22.631944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.631912    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497598bb-c03c-4f3f-91eb-de081b18b984" path="/var/lib/kubelet/pods/497598bb-c03c-4f3f-91eb-de081b18b984/volumes"
Oct 30 20:06:22.964218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964185    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"]
Oct 30 20:06:22.964791 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964774    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d10dbe94-800c-4348-8371-f8a65c9f03da" containerName="maas-api"
Oct 30 20:06:22.964848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964794    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10dbe94-800c-4348-8371-f8a65c9f03da" containerName="maas-api"
Oct 30 20:06:22.964848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964816    5085 cpu_manager.go:414] "containerMap: removing container" podUID="497598bb-c03c-4f3f-91eb-de081b18b984" containerName="payload-pre-processing"
Oct 30 20:06:22.964848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964822    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497598bb-c03c-4f3f-91eb-de081b18b984" containerName="payload-pre-processing"
Oct 30 20:06:22.964848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964847    5085 cpu_manager.go:414] "containerMap: removing container" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy"
Oct 30 20:06:22.964975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964853    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy"
Oct 30 20:06:22.964975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964864    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" containerName="maas-api"
Oct 30 20:06:22.964975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964874    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" containerName="maas-api"
Oct 30 20:06:22.964975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964891    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy"
Oct 30 20:06:22.964975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964895    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy"
Oct 30 20:06:22.965195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964979    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cee3d9-2472-45a8-adb9-11a876d6ea21" containerName="istio-proxy"
Oct 30 20:06:22.965195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964990    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7da320-725c-42b5-b6a0-f2d024c8a62c" containerName="maas-api"
Oct 30 20:06:22.965195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.964998    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5c97c8-7a5d-4163-ac58-8bcccb4f2777" containerName="istio-proxy"
Oct 30 20:06:22.965195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.965006    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10dbe94-800c-4348-8371-f8a65c9f03da" containerName="maas-api"
Oct 30 20:06:22.965195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.965014    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="497598bb-c03c-4f3f-91eb-de081b18b984" containerName="payload-pre-processing"
Oct 30 20:06:22.969873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:22.969855    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.066231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.066201    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:23.066231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.066212    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:06:23.069162 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.069142    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"]
Oct 30 20:06:23.090207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.090178    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.090374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.090238    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gnt\" (UniqueName: \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.090374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.090343    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.192015 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.191975    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.192170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.192055    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gnt\" (UniqueName: \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.192170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.192089    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.192678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.192654    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.194334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.194312    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.250186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.250114    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gnt\" (UniqueName: \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\") pod \"maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.283260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.283232    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:23.658247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.658218    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"]
Oct 30 20:06:23.660183 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:23.660150    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ac86fc_a6f6_465f_89b5_74654767d706.slice/crio-86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706 WatchSource:0}: Error finding container 86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706: Status 404 returned error can't find the container with id 86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706
Oct 30 20:06:23.899222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:23.899191    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr" event={"ID":"c9ac86fc-a6f6-465f-89b5-74654767d706","Type":"ContainerStarted","Data":"86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706"}
Oct 30 20:06:24.907724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:24.907632    5085 generic.go:356] "Generic (PLEG): container finished" podID="c9ac86fc-a6f6-465f-89b5-74654767d706" containerID="1d78f4fda9727b4fce37a3023381920c69bf8b4f0a06da6793f34ffb6e2d6e90" exitCode=0
Oct 30 20:06:24.908097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:24.907716    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr" event={"ID":"c9ac86fc-a6f6-465f-89b5-74654767d706","Type":"ContainerDied","Data":"1d78f4fda9727b4fce37a3023381920c69bf8b4f0a06da6793f34ffb6e2d6e90"}
Oct 30 20:06:26.065718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.065691    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:26.130178 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.130146    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\") pod \"c9ac86fc-a6f6-465f-89b5-74654767d706\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") "
Oct 30 20:06:26.130343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.130229    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\") pod \"c9ac86fc-a6f6-465f-89b5-74654767d706\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") "
Oct 30 20:06:26.130343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.130273    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\" (UniqueName: \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\") pod \"c9ac86fc-a6f6-465f-89b5-74654767d706\" (UID: \"c9ac86fc-a6f6-465f-89b5-74654767d706\") "
Oct 30 20:06:26.130500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.130476    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca" pod "c9ac86fc-a6f6-465f-89b5-74654767d706" (UID: "c9ac86fc-a6f6-465f-89b5-74654767d706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:26.130789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.130769    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:26.132409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.132381    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt" pod "c9ac86fc-a6f6-465f-89b5-74654767d706" (UID: "c9ac86fc-a6f6-465f-89b5-74654767d706"). InnerVolumeSpecName "kube-api-access-t7gnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:26.132495 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.132462    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca" pod "c9ac86fc-a6f6-465f-89b5-74654767d706" (UID: "c9ac86fc-a6f6-465f-89b5-74654767d706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:26.232097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.232062    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ac86fc-a6f6-465f-89b5-74654767d706-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:26.232097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.232091    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gnt\" (UniqueName: \"kubernetes.io/projected/c9ac86fc-a6f6-465f-89b5-74654767d706-kube-api-access-t7gnt\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:06:26.922815 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.922788    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"
Oct 30 20:06:26.922968 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.922806    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr" event={"ID":"c9ac86fc-a6f6-465f-89b5-74654767d706","Type":"ContainerDied","Data":"86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706"}
Oct 30 20:06:26.922968 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:26.922869    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86fa1b42fc7a53249507c7f1de7c8222bb3686e2a0ee2040d12996daf274e706"
Oct 30 20:06:30.425903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.425868    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:06:30.426398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.426385    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c9ac86fc-a6f6-465f-89b5-74654767d706" containerName="revoke-keys"
Oct 30 20:06:30.426443 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.426399    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ac86fc-a6f6-465f-89b5-74654767d706" containerName="revoke-keys"
Oct 30 20:06:30.426518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.426509    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ac86fc-a6f6-465f-89b5-74654767d706" containerName="revoke-keys"
Oct 30 20:06:30.430238 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.430219    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.482753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.482725    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-84ad63-openshift-default-dockercfg-px9bp\""
Oct 30 20:06:30.533121 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.533086    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:06:30.594545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594514    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594623    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594697    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594716    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w7bv\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594754    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594851    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594937 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594884    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.594937 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.594906    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.595072 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.595041    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696383    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696426    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696444    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696550    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696641    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696719    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696828    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696875    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.696957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696900    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w7bv\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.697126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.696981    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.697126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.697065    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.697260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.697239    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.697324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.697297    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.697384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.697362    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.698918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.698896    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.699279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.699259    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.756500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.756471    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:30.756943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:30.756916    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w7bv\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\") pod \"e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") " pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:31.044140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.044054    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:31.379259 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.379172    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:06:31.384456 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.384429    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.439425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.434386    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-389c4e-openshift-default-dockercfg-kmrqw\""
Oct 30 20:06:31.476486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.476455    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:06:31.521361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.520681    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:06:31.531805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.531632    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.531805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.531712    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk46\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.531805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.531752    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.531995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.531894    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.531995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.531960    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.532112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.532031    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.532112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.532066    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.532216 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.532120    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.532274 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.532248    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.535975 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:31.535707    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa2dc8f_8452_4073_a50b_2f4e00abcce8.slice/crio-332a8d0d39ad01c6d051ae5d9ffe06d24533bec1737c4ee43d124fd0db0c2d1e WatchSource:0}: Error finding container 332a8d0d39ad01c6d051ae5d9ffe06d24533bec1737c4ee43d124fd0db0c2d1e: Status 404 returned error can't find the container with id 332a8d0d39ad01c6d051ae5d9ffe06d24533bec1737c4ee43d124fd0db0c2d1e
Oct 30 20:06:31.539885 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.539838    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:31.540046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.539933    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:31.540046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.539975    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:31.633328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633244    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633305    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633361    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633388    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633422    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633522    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633600    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633715    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk46\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633761    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.633827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633769    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.634054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.633922    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.634285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.634264    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.634384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.634313    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.634554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.634527    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.636140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.636112    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.636372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.636355    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.705107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.705072    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.709224 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.709193    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk46\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\") pod \"e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") " pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:31.957607 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.957535    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerStarted","Data":"ad898faba2b2dcdff05fbc9e066e846f3b1cc552bc3a53fd7edfcb672ee3417e"}
Oct 30 20:06:31.957791 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.957627    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerStarted","Data":"332a8d0d39ad01c6d051ae5d9ffe06d24533bec1737c4ee43d124fd0db0c2d1e"}
Oct 30 20:06:31.999044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:31.999015    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:32.044433 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.044392    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:32.050817 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.050789    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:32.079657 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.079602    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podStartSLOduration=2.079582857 podStartE2EDuration="2.079582857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:32.07631857 +0000 UTC m=+2501.580722330" watchObservedRunningTime="2026-10-30 20:06:32.079582857 +0000 UTC m=+2501.583986606"
Oct 30 20:06:32.426360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.426329    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:06:32.427184 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:32.427156    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbac9a96_319e_4a4b_9542_6c387618c6f6.slice/crio-89e4ddc3712491ab0111b138a5ae06256f6d6d5e80626842167c31e713807e2e WatchSource:0}: Error finding container 89e4ddc3712491ab0111b138a5ae06256f6d6d5e80626842167c31e713807e2e: Status 404 returned error can't find the container with id 89e4ddc3712491ab0111b138a5ae06256f6d6d5e80626842167c31e713807e2e
Oct 30 20:06:32.430071 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.430038    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:32.430144 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.430113    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:32.430194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.430143    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:06:32.966026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.965982    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerStarted","Data":"71073c27b1578f5628d9c418cdc05777a49c978e42b14a9245ccbb41a195fc34"}
Oct 30 20:06:32.966473 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.966057    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerStarted","Data":"89e4ddc3712491ab0111b138a5ae06256f6d6d5e80626842167c31e713807e2e"}
Oct 30 20:06:32.966473 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.966372    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:32.967699 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.967674    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:06:32.999826 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:32.999784    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:33.081128 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:33.081037    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podStartSLOduration=2.080992738 podStartE2EDuration="2.080992738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:33.080149338 +0000 UTC m=+2502.584553086" watchObservedRunningTime="2026-10-30 20:06:33.080992738 +0000 UTC m=+2502.585396488"
Oct 30 20:06:34.005006 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:34.004974    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:34.984410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:34.984375    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:34.985469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:34.985438    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:06:36.183068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:36.183034    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:06:38.657732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.657705    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:06:38.662877 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.662852    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.711217 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.711187    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8j4s\" (UniqueName: \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.711357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.711270    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.713119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.713094    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-84ad63\""
Oct 30 20:06:38.774300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.774264    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-84ad63-dockercfg-f6htg\""
Oct 30 20:06:38.806719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.806681    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:06:38.813022 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.812988    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j4s\" (UniqueName: \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.813194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.813098    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.813964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.813938    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.824490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.824459    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:06:38.828814 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.828796    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:38.872263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.872233    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j4s\" (UniqueName: \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\") pod \"payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:38.872472 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.872445    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:06:38.914170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.914101    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:38.914316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.914179    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mtn\" (UniqueName: \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:38.977465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:38.977427    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:39.014835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.014806    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:39.015000 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.014922    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mtn\" (UniqueName: \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:39.015773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.015747    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:39.075388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.074960    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mtn\" (UniqueName: \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\") pod \"payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:39.141593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.141546    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:39.349872 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:39.349839    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f08ea7a_a4f7_41bc_a450_636a96ffa72c.slice/crio-64ca8edd17922a9585a0d39df078640ca86ef2fc7e1fe143d3fcce5c701a9732 WatchSource:0}: Error finding container 64ca8edd17922a9585a0d39df078640ca86ef2fc7e1fe143d3fcce5c701a9732: Status 404 returned error can't find the container with id 64ca8edd17922a9585a0d39df078640ca86ef2fc7e1fe143d3fcce5c701a9732
Oct 30 20:06:39.350416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.350357    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:06:39.512407 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:39.512383    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:06:39.513527 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:39.513500    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6cd9070_9c4b_4d84_8fde_96701967ea36.slice/crio-6ba2621e05aecb0914784d9245c6d177e9c772aa4e2c71097cd1a1f02072c937 WatchSource:0}: Error finding container 6ba2621e05aecb0914784d9245c6d177e9c772aa4e2c71097cd1a1f02072c937: Status 404 returned error can't find the container with id 6ba2621e05aecb0914784d9245c6d177e9c772aa4e2c71097cd1a1f02072c937
Oct 30 20:06:40.017714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.017670    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" event={"ID":"d6cd9070-9c4b-4d84-8fde-96701967ea36","Type":"ContainerStarted","Data":"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"}
Oct 30 20:06:40.018145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.017740    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" event={"ID":"d6cd9070-9c4b-4d84-8fde-96701967ea36","Type":"ContainerStarted","Data":"6ba2621e05aecb0914784d9245c6d177e9c772aa4e2c71097cd1a1f02072c937"}
Oct 30 20:06:40.018145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.017774    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:40.020155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.020119    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" event={"ID":"9f08ea7a-a4f7-41bc-a450-636a96ffa72c","Type":"ContainerStarted","Data":"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"}
Oct 30 20:06:40.020305 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.020232    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" event={"ID":"9f08ea7a-a4f7-41bc-a450-636a96ffa72c","Type":"ContainerStarted","Data":"64ca8edd17922a9585a0d39df078640ca86ef2fc7e1fe143d3fcce5c701a9732"}
Oct 30 20:06:40.020678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.020652    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:40.132574 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.132513    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" podStartSLOduration=2.132498162 podStartE2EDuration="2.132498162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:40.127984386 +0000 UTC m=+2509.632388133" watchObservedRunningTime="2026-10-30 20:06:40.132498162 +0000 UTC m=+2509.636901939"
Oct 30 20:06:40.240768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:40.240712    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" podStartSLOduration=2.240698334 podStartE2EDuration="2.240698334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:40.239169047 +0000 UTC m=+2509.743572809" watchObservedRunningTime="2026-10-30 20:06:40.240698334 +0000 UTC m=+2509.745102082"
Oct 30 20:06:43.662614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.662580    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:06:43.666801 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.666781    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.725977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.725948    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:06:43.796912 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.796881    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-389c4e\""
Oct 30 20:06:43.797117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.797037    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-389c4e-dockercfg-kwmr8\""
Oct 30 20:06:43.798702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.798673    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9zjl\" (UniqueName: \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.798830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.798805    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.846846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.846804    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:06:43.850923 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.850901    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:43.874095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.874067    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:06:43.900149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.900121    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9zjl\" (UniqueName: \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.900333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.900227    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.901059 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.901033    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.960119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.960090    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9zjl\" (UniqueName: \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\") pod \"payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:43.982020 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:43.981995    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:44.001359 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.001334    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.001537 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.001522    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56csn\" (UniqueName: \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.102770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.102730    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.102970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.102859    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56csn\" (UniqueName: \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.103520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.103491    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.164158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.164118    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56csn\" (UniqueName: \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\") pod \"payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g\" (UID: \"c7918962-8970-4916-960b-a31341090029\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.166033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.166009    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:44.359969 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.359935    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:06:44.360370 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:44.360343    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec296583_5915_4c7a_8ea7_96d3a7f6647d.slice/crio-0f0fd563843d3fd0c1bc3f1668c7f09fd543116e8b44de7e94242390af7710e9 WatchSource:0}: Error finding container 0f0fd563843d3fd0c1bc3f1668c7f09fd543116e8b44de7e94242390af7710e9: Status 404 returned error can't find the container with id 0f0fd563843d3fd0c1bc3f1668c7f09fd543116e8b44de7e94242390af7710e9
Oct 30 20:06:44.549909 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:44.549878    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:06:44.551183 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:44.551159    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7918962_8970_4916_960b_a31341090029.slice/crio-abf3e471f08d01f9f315e1e07aa0c0c0cfce3eb11ab93d8b7f3384178931e16d WatchSource:0}: Error finding container abf3e471f08d01f9f315e1e07aa0c0c0cfce3eb11ab93d8b7f3384178931e16d: Status 404 returned error can't find the container with id abf3e471f08d01f9f315e1e07aa0c0c0cfce3eb11ab93d8b7f3384178931e16d
Oct 30 20:06:45.068641 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.068598    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" event={"ID":"c7918962-8970-4916-960b-a31341090029","Type":"ContainerStarted","Data":"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"}
Oct 30 20:06:45.069167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.068658    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" event={"ID":"c7918962-8970-4916-960b-a31341090029","Type":"ContainerStarted","Data":"abf3e471f08d01f9f315e1e07aa0c0c0cfce3eb11ab93d8b7f3384178931e16d"}
Oct 30 20:06:45.069167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.068882    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:45.070339 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.070314    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" event={"ID":"ec296583-5915-4c7a-8ea7-96d3a7f6647d","Type":"ContainerStarted","Data":"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"}
Oct 30 20:06:45.070477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.070359    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" event={"ID":"ec296583-5915-4c7a-8ea7-96d3a7f6647d","Type":"ContainerStarted","Data":"0f0fd563843d3fd0c1bc3f1668c7f09fd543116e8b44de7e94242390af7710e9"}
Oct 30 20:06:45.070477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.070422    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:45.182905 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.182837    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" podStartSLOduration=2.182818702 podStartE2EDuration="2.182818702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:45.179342814 +0000 UTC m=+2514.683746563" watchObservedRunningTime="2026-10-30 20:06:45.182818702 +0000 UTC m=+2514.687222451"
Oct 30 20:06:45.292164 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:45.292111    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" podStartSLOduration=2.292094386 podStartE2EDuration="2.292094386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:45.289608214 +0000 UTC m=+2514.794011957" watchObservedRunningTime="2026-10-30 20:06:45.292094386 +0000 UTC m=+2514.796498145"
Oct 30 20:06:49.941164 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:49.941130    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:06:49.945675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:49.945653    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:49.998973 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:49.998943    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f61bae-serving-cert\""
Oct 30 20:06:50.003209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.003179    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:06:50.071449 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.071414    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8c2j\" (UniqueName: \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.071663 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.071505    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.071663 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.071606    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.071663 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.071629    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.173019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.172981    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.173204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.173032    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.173204 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:50.173113    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-serving-cert: secret "maas-api-e2e-praxis-a-f61bae-serving-cert" not found
Oct 30 20:06:50.173204 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:50.173180    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls podName:995bc9bc-93f8-44eb-840a-70f3a133ce88 nodeName:}" failed. No retries permitted until 2026-10-30 20:06:50.673163132 +0000 UTC m=+2520.177566880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls") pod "maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" (UID: "995bc9bc-93f8-44eb-840a-70f3a133ce88") : secret "maas-api-e2e-praxis-a-f61bae-serving-cert" not found
Oct 30 20:06:50.173372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.173220    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8c2j\" (UniqueName: \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.173372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.173251    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.173913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.173896    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.175420 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.175403    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.233341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.233251    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8c2j\" (UniqueName: \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.679684 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.679655    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.682014 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.681995    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:50.863257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:50.863223    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:51.029525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:51.029455    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:06:51.029897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:51.029528    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:06:51.303247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:51.303220    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:06:51.304989 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:06:51.304961    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod995bc9bc_93f8_44eb_840a_70f3a133ce88.slice/crio-509f19c39b0a809a608a3d079bda0403280382bc0ca8556a8aa28af0540f512e WatchSource:0}: Error finding container 509f19c39b0a809a608a3d079bda0403280382bc0ca8556a8aa28af0540f512e: Status 404 returned error can't find the container with id 509f19c39b0a809a608a3d079bda0403280382bc0ca8556a8aa28af0540f512e
Oct 30 20:06:51.939860 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:06:51.939827    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:06:52.117202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:52.117167    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" event={"ID":"995bc9bc-93f8-44eb-840a-70f3a133ce88","Type":"ContainerStarted","Data":"509f19c39b0a809a608a3d079bda0403280382bc0ca8556a8aa28af0540f512e"}
Oct 30 20:06:53.125511 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:53.125477    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" event={"ID":"995bc9bc-93f8-44eb-840a-70f3a133ce88","Type":"ContainerStarted","Data":"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"}
Oct 30 20:06:53.126036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:53.125549    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:06:53.247749 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:53.247691    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" podStartSLOduration=3.280944001 podStartE2EDuration="4.247677912s" totalImagesPullingTime="966.733911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:53.244242909 +0000 UTC m=+2522.748646656" watchObservedRunningTime="2026-10-30 20:06:53.247677912 +0000 UTC m=+2522.752081658"
Oct 30 20:06:56.080364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:56.080333    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:06:56.080856 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:56.080404    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:06:59.138803 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:06:59.138773    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:07:07.176117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.176082    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:07:07.176664 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.176298    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" podUID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" containerName="maas-api" containerID="cri-o://510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7" gracePeriod=30
Oct 30 20:07:07.454655 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.454630    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:07:07.597698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.597664    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") pod \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") "
Oct 30 20:07:07.597913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.597765    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\" (UniqueName: \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\") pod \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") "
Oct 30 20:07:07.597913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.597819    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\") pod \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") "
Oct 30 20:07:07.597913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.597882    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\") pod \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\" (UID: \"91dfdce5-e842-448a-8aab-0bfd7291c8e0\") "
Oct 30 20:07:07.598619 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.598546    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca" pod "91dfdce5-e842-448a-8aab-0bfd7291c8e0" (UID: "91dfdce5-e842-448a-8aab-0bfd7291c8e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:07.601232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.601200    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls" pod "91dfdce5-e842-448a-8aab-0bfd7291c8e0" (UID: "91dfdce5-e842-448a-8aab-0bfd7291c8e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:07.613606 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.603210    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs" pod "91dfdce5-e842-448a-8aab-0bfd7291c8e0" (UID: "91dfdce5-e842-448a-8aab-0bfd7291c8e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:07.613606 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.603248    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6" pod "91dfdce5-e842-448a-8aab-0bfd7291c8e0" (UID: "91dfdce5-e842-448a-8aab-0bfd7291c8e0"). InnerVolumeSpecName "kube-api-access-j2fv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:07.655152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.655119    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:07:07.655334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.655313    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" podUID="73c1bf14-656c-4e36-9b68-94c2b23c2997" containerName="payload-processing" containerID="cri-o://df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2" gracePeriod=30
Oct 30 20:07:07.699893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.699814    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:07.699893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.699842    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2fv6\" (UniqueName: \"kubernetes.io/projected/91dfdce5-e842-448a-8aab-0bfd7291c8e0-kube-api-access-j2fv6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:07.699893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.699852    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91dfdce5-e842-448a-8aab-0bfd7291c8e0-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:07.699893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.699864    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91dfdce5-e842-448a-8aab-0bfd7291c8e0-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:07.921350 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:07.921322    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:07:08.003404 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.003329    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\") pod \"73c1bf14-656c-4e36-9b68-94c2b23c2997\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") "
Oct 30 20:07:08.003404 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.003390    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\") pod \"73c1bf14-656c-4e36-9b68-94c2b23c2997\" (UID: \"73c1bf14-656c-4e36-9b68-94c2b23c2997\") "
Oct 30 20:07:08.003820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.003789    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume" pod "73c1bf14-656c-4e36-9b68-94c2b23c2997" (UID: "73c1bf14-656c-4e36-9b68-94c2b23c2997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:08.003952 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.003881    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c1bf14-656c-4e36-9b68-94c2b23c2997-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:08.005417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.005395    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp" pod "73c1bf14-656c-4e36-9b68-94c2b23c2997" (UID: "73c1bf14-656c-4e36-9b68-94c2b23c2997"). InnerVolumeSpecName "kube-api-access-xbrcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:08.104895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.104861    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrcp\" (UniqueName: \"kubernetes.io/projected/73c1bf14-656c-4e36-9b68-94c2b23c2997-kube-api-access-xbrcp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:08.245313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.245271    5085 generic.go:356] "Generic (PLEG): container finished" podID="73c1bf14-656c-4e36-9b68-94c2b23c2997" containerID="df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2" exitCode=0
Oct 30 20:07:08.245313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.245311    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"
Oct 30 20:07:08.245313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.245307    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" event={"ID":"73c1bf14-656c-4e36-9b68-94c2b23c2997","Type":"ContainerDied","Data":"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"}
Oct 30 20:07:08.245970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.245371    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj" event={"ID":"73c1bf14-656c-4e36-9b68-94c2b23c2997","Type":"ContainerDied","Data":"84655a70b4d5eca6dffe2f9e336f17742ec52015730a05bc71cc101f0663cf08"}
Oct 30 20:07:08.245970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.245401    5085 scope.go:122] "RemoveContainer" containerID="df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"
Oct 30 20:07:08.247171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.247151    5085 generic.go:356] "Generic (PLEG): container finished" podID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" containerID="510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7" exitCode=0
Oct 30 20:07:08.247292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.247190    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" event={"ID":"91dfdce5-e842-448a-8aab-0bfd7291c8e0","Type":"ContainerDied","Data":"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"}
Oct 30 20:07:08.247292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.247204    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"
Oct 30 20:07:08.247292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.247227    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln" event={"ID":"91dfdce5-e842-448a-8aab-0bfd7291c8e0","Type":"ContainerDied","Data":"ca9e7297fe448f0518364352741956bb923406963461cf2d19dce4ee432a5562"}
Oct 30 20:07:08.257131 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.257112    5085 scope.go:122] "RemoveContainer" containerID="df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"
Oct 30 20:07:08.257402 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:08.257381    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2\": container with ID starting with df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2 not found: ID does not exist" containerID="df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"
Oct 30 20:07:08.257453 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.257411    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2"} err="failed to get container status \"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2\": rpc error: code = NotFound desc = could not find container \"df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2\": container with ID starting with df8f32fa8a30c827ab8b9e86a571b60a4512be29c9e3ef58b0302797de3197b2 not found: ID does not exist"
Oct 30 20:07:08.257453 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.257428    5085 scope.go:122] "RemoveContainer" containerID="510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"
Oct 30 20:07:08.267428 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.267404    5085 scope.go:122] "RemoveContainer" containerID="510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"
Oct 30 20:07:08.267718 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:08.267702    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7\": container with ID starting with 510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7 not found: ID does not exist" containerID="510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"
Oct 30 20:07:08.267789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.267724    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7"} err="failed to get container status \"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7\": rpc error: code = NotFound desc = could not find container \"510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7\": container with ID starting with 510d95ec12a3dab30d95706a064b7cd68465851a04b694c7f3d29ef66b563cc7 not found: ID does not exist"
Oct 30 20:07:08.403131 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.403087    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:07:08.410013 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.409986    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2d2ca9-6f4698d568-lhlsj"]
Oct 30 20:07:08.514008 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.513930    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:07:08.522058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.522030    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2d2ca9-5d675b4c76-7ncln"]
Oct 30 20:07:08.631530 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.631498    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c1bf14-656c-4e36-9b68-94c2b23c2997" path="/var/lib/kubelet/pods/73c1bf14-656c-4e36-9b68-94c2b23c2997/volumes"
Oct 30 20:07:08.631958 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:08.631943    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" path="/var/lib/kubelet/pods/91dfdce5-e842-448a-8aab-0bfd7291c8e0/volumes"
Oct 30 20:07:10.039438 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:10.039406    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"]
Oct 30 20:07:10.086449 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:10.086416    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2d2ca9-pcpdr"]
Oct 30 20:07:10.633383 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:10.633351    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ac86fc-a6f6-465f-89b5-74654767d706" path="/var/lib/kubelet/pods/c9ac86fc-a6f6-465f-89b5-74654767d706/volumes"
Oct 30 20:07:18.495320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.495285    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"]
Oct 30 20:07:18.495918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.495903    5085 cpu_manager.go:414] "containerMap: removing container" podUID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" containerName="maas-api"
Oct 30 20:07:18.495967 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.495919    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" containerName="maas-api"
Oct 30 20:07:18.496000 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.495986    5085 cpu_manager.go:414] "containerMap: removing container" podUID="73c1bf14-656c-4e36-9b68-94c2b23c2997" containerName="payload-processing"
Oct 30 20:07:18.496000 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.495992    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c1bf14-656c-4e36-9b68-94c2b23c2997" containerName="payload-processing"
Oct 30 20:07:18.496112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.496102    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c1bf14-656c-4e36-9b68-94c2b23c2997" containerName="payload-processing"
Oct 30 20:07:18.496145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.496116    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="91dfdce5-e842-448a-8aab-0bfd7291c8e0" containerName="maas-api"
Oct 30 20:07:18.499892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.499874    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.542931 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.542905    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.543080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.542949    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwqd\" (UniqueName: \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.543080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.543045    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.550016 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.549994    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:07:18.552136 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.552118    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:18.600390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.600361    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"]
Oct 30 20:07:18.644147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.644111    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.644315 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.644167    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwqd\" (UniqueName: \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.644372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.644310    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.645065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.645047    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.646684 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.646663    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.703133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.703101    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwqd\" (UniqueName: \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\") pod \"maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:18.813555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:18.813471    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:19.189824 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:19.189794    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"]
Oct 30 20:07:19.190754 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:07:19.190725    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186bbf5d_3506_4f3e_b43e_bb51dd01eb69.slice/crio-82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3 WatchSource:0}: Error finding container 82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3: Status 404 returned error can't find the container with id 82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3
Oct 30 20:07:19.337785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:19.337748    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg" event={"ID":"186bbf5d-3506-4f3e-b43e-bb51dd01eb69","Type":"ContainerStarted","Data":"82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3"}
Oct 30 20:07:20.345536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:20.345503    5085 generic.go:356] "Generic (PLEG): container finished" podID="186bbf5d-3506-4f3e-b43e-bb51dd01eb69" containerID="c62d1978beec8165ac6672d28361576e2be76c1cb3eb79a6bde8779de9eb7f35" exitCode=0
Oct 30 20:07:20.345966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:20.345545    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg" event={"ID":"186bbf5d-3506-4f3e-b43e-bb51dd01eb69","Type":"ContainerDied","Data":"c62d1978beec8165ac6672d28361576e2be76c1cb3eb79a6bde8779de9eb7f35"}
Oct 30 20:07:21.507775 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.507750    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:21.579768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.579733    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\" (UniqueName: \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\") pod \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") "
Oct 30 20:07:21.579942 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.579924    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\") pod \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") "
Oct 30 20:07:21.579985 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.579975    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\") pod \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\" (UID: \"186bbf5d-3506-4f3e-b43e-bb51dd01eb69\") "
Oct 30 20:07:21.580273 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.580250    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca" pod "186bbf5d-3506-4f3e-b43e-bb51dd01eb69" (UID: "186bbf5d-3506-4f3e-b43e-bb51dd01eb69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:21.580440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.580425    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:21.581940 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.581918    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd" pod "186bbf5d-3506-4f3e-b43e-bb51dd01eb69" (UID: "186bbf5d-3506-4f3e-b43e-bb51dd01eb69"). InnerVolumeSpecName "kube-api-access-xwwqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:21.582046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.581975    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca" pod "186bbf5d-3506-4f3e-b43e-bb51dd01eb69" (UID: "186bbf5d-3506-4f3e-b43e-bb51dd01eb69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:21.681181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.681154    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:21.681181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:21.681177    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwqd\" (UniqueName: \"kubernetes.io/projected/186bbf5d-3506-4f3e-b43e-bb51dd01eb69-kube-api-access-xwwqd\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:07:21.941890 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:21.941804    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:07:22.359080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:22.358990    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"
Oct 30 20:07:22.359080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:22.359014    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg" event={"ID":"186bbf5d-3506-4f3e-b43e-bb51dd01eb69","Type":"ContainerDied","Data":"82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3"}
Oct 30 20:07:22.359302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:22.359090    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82bdc4dc06e31003d92fc05ade84be1c2383c766b16a8662156c31a307d528f3"
Oct 30 20:07:27.724490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:27.724409    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:07:28.006518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:28.006441    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:07:32.016399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:32.016360    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:07:33.121794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.121764    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:07:33.122346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.122332    5085 cpu_manager.go:414] "containerMap: removing container" podUID="186bbf5d-3506-4f3e-b43e-bb51dd01eb69" containerName="revoke-keys"
Oct 30 20:07:33.122415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.122347    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186bbf5d-3506-4f3e-b43e-bb51dd01eb69" containerName="revoke-keys"
Oct 30 20:07:33.122454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.122436    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="186bbf5d-3506-4f3e-b43e-bb51dd01eb69" containerName="revoke-keys"
Oct 30 20:07:33.127281 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.127265    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216215    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216412 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216353    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216421    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216440    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ghz6\" (UniqueName: \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216604 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216554    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.216664 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.216655    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.217536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.217523    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eaa6d2\"/\"default-dockercfg-fln79\""
Oct 30 20:07:33.225410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.225385    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eaa6d2\"/\"ipp-route-eaa6d2-kserve-self-signed-certs\""
Oct 30 20:07:33.225410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.225390    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eaa6d2\"/\"kube-root-ca.crt\""
Oct 30 20:07:33.225601 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.225469    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eaa6d2\"/\"openshift-service-ca.crt\""
Oct 30 20:07:33.228831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.228811    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:07:33.317458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317424    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317457    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ghz6\" (UniqueName: \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317482    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317518    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317555    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317654    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.317960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317937    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.318002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.317985    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.318044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.318028    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.319845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.319822    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.320269 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.320246    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.380694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.380623    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ghz6\" (UniqueName: \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\") pod \"ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") " pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.441057 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.441019    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:33.816688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:33.816657    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:07:33.818383 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:07:33.818356    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac00388d_39fd_4e68_a249_3430e4591667.slice/crio-2ee689b7f5d84ddf40c5dd441b4cf3fe713530462f3b85f7e54fdd24015f7c08 WatchSource:0}: Error finding container 2ee689b7f5d84ddf40c5dd441b4cf3fe713530462f3b85f7e54fdd24015f7c08: Status 404 returned error can't find the container with id 2ee689b7f5d84ddf40c5dd441b4cf3fe713530462f3b85f7e54fdd24015f7c08
Oct 30 20:07:34.450716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.450676    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" event={"ID":"ac00388d-39fd-4e68-a249-3430e4591667","Type":"ContainerStarted","Data":"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"}
Oct 30 20:07:34.451159 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.450728    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" event={"ID":"ac00388d-39fd-4e68-a249-3430e4591667","Type":"ContainerStarted","Data":"2ee689b7f5d84ddf40c5dd441b4cf3fe713530462f3b85f7e54fdd24015f7c08"}
Oct 30 20:07:34.562620 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.562554    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" podStartSLOduration=1.5625403690000002 podStartE2EDuration="1.562540369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:34.561826221 +0000 UTC m=+2564.066229969" watchObservedRunningTime="2026-10-30 20:07:34.562540369 +0000 UTC m=+2564.066944133"
Oct 30 20:07:34.920356 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.920327    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:07:34.924820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.924793    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:34.977698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.977622    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-84ad63-serving-cert\""
Oct 30 20:07:34.982167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:34.982139    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:07:35.037177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.037142    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.037344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.037229    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.037344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.037308    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.037532 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.037510    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbswg\" (UniqueName: \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.138411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.138372    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbswg\" (UniqueName: \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.138638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.138468    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.138638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.138491    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.138638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.138529    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.138786 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:35.138640    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-serving-cert: secret "maas-api-e2e-praxis-a-84ad63-serving-cert" not found
Oct 30 20:07:35.138786 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:35.138709    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls podName:f22402bd-50b4-46fb-b209-10992ba4decd nodeName:}" failed. No retries permitted until 2026-10-30 20:07:35.638686979 +0000 UTC m=+2565.143090705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls") pod "maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" (UID: "f22402bd-50b4-46fb-b209-10992ba4decd") : secret "maas-api-e2e-praxis-a-84ad63-serving-cert" not found
Oct 30 20:07:35.139218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.139193    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.141015 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.140992    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.199585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.199502    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbswg\" (UniqueName: \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.441436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.441403    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:35.643599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.643505    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.645858 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.645837    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") pod \"maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:35.839243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:35.839207    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:36.212499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:36.212469    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:07:36.213807 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:07:36.213772    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22402bd_50b4_46fb_b209_10992ba4decd.slice/crio-0757a3925a965eab1ac47c7057472f6cf0b79e96e61da5ecc69a8ff178e4e839 WatchSource:0}: Error finding container 0757a3925a965eab1ac47c7057472f6cf0b79e96e61da5ecc69a8ff178e4e839: Status 404 returned error can't find the container with id 0757a3925a965eab1ac47c7057472f6cf0b79e96e61da5ecc69a8ff178e4e839
Oct 30 20:07:36.465882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:36.465793    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" event={"ID":"f22402bd-50b4-46fb-b209-10992ba4decd","Type":"ContainerStarted","Data":"0757a3925a965eab1ac47c7057472f6cf0b79e96e61da5ecc69a8ff178e4e839"}
Oct 30 20:07:37.473711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:37.473622    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" event={"ID":"f22402bd-50b4-46fb-b209-10992ba4decd","Type":"ContainerStarted","Data":"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"}
Oct 30 20:07:37.474088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:37.473824    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:37.635423 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:37.635368    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" podStartSLOduration=2.678153073 podStartE2EDuration="3.635353023s" totalImagesPullingTime="957.19995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:37.633272792 +0000 UTC m=+2567.137676539" watchObservedRunningTime="2026-10-30 20:07:37.635353023 +0000 UTC m=+2567.139756791"
Oct 30 20:07:40.043544 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.043512    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:07:40.047880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.047860    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.147415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.147391    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-389c4e-serving-cert\""
Oct 30 20:07:40.150748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.150722    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:07:40.197927 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.197887    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcmg\" (UniqueName: \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.198129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.198041    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.198206 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.198127    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.198266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.198249    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.300342 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.300202    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcmg\" (UniqueName: \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.300539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.300373    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.300539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.300430    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.300966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.300937    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.304771 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:40.301171    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-serving-cert: secret "maas-api-e2e-praxis-a-389c4e-serving-cert" not found
Oct 30 20:07:40.304771 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:40.301244    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls podName:4a237a18-63a5-4fa9-a420-fe9a1d49d586 nodeName:}" failed. No retries permitted until 2026-10-30 20:07:40.801223467 +0000 UTC m=+2570.305627210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls") pod "maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" (UID: "4a237a18-63a5-4fa9-a420-fe9a1d49d586") : secret "maas-api-e2e-praxis-a-389c4e-serving-cert" not found
Oct 30 20:07:40.304771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.301639    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.304771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.303849    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.359693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.359654    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcmg\" (UniqueName: \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.807906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.807874    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.810287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.810257    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") pod \"maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:40.961369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:40.961325    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:41.336241 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:41.336200    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:07:41.503357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:41.503320    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" event={"ID":"4a237a18-63a5-4fa9-a420-fe9a1d49d586","Type":"ContainerStarted","Data":"f285d3100ee83f43c3336b17dec0f6bcf594af7ff9a2acbf1aec492cde09acde"}
Oct 30 20:07:42.511962 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:42.511876    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" event={"ID":"4a237a18-63a5-4fa9-a420-fe9a1d49d586","Type":"ContainerStarted","Data":"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"}
Oct 30 20:07:42.512352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:42.512062    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:42.626666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:42.626610    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" podStartSLOduration=2.774337262 podStartE2EDuration="3.626593158s" totalImagesPullingTime="852.255896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:42.623447067 +0000 UTC m=+2572.127850816" watchObservedRunningTime="2026-10-30 20:07:42.626593158 +0000 UTC m=+2572.130996910"
Oct 30 20:07:43.441888 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:43.441857    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:43.455098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:43.455072    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:43.486614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:43.486589    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:07:43.531173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:43.531145    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:07:48.526480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:07:48.526451    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:07:51.942125 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:07:51.942086    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:08:00.678732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.678696    5085 generic.go:356] "Generic (PLEG): container finished" podID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerID="ad898faba2b2dcdff05fbc9e066e846f3b1cc552bc3a53fd7edfcb672ee3417e" exitCode=0
Oct 30 20:08:00.679163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.678741    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerDied","Data":"ad898faba2b2dcdff05fbc9e066e846f3b1cc552bc3a53fd7edfcb672ee3417e"}
Oct 30 20:08:00.679163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.679142    5085 scope.go:122] "RemoveContainer" containerID="ad898faba2b2dcdff05fbc9e066e846f3b1cc552bc3a53fd7edfcb672ee3417e"
Oct 30 20:08:00.680761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.680734    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:00.680827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.680802    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:00.680867 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:00.680833    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:01.689702 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:01.689656    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerStarted","Data":"aea2b390d1d2af768ed441f51e32256e170cc2e3020e33bb988f282c48f71725"}
Oct 30 20:08:02.044673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:02.044589    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:08:02.045948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:02.045918    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 20:08:02.046096 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:02.045987    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 20:08:02.979703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:02.979667    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:08:03.045198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:03.045160    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 20:08:03.045385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:03.045224    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 20:08:04.044651 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:04.044617    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 20:08:04.045074 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:04.044691    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 20:08:05.045104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:05.045069    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 20:08:05.045510 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:05.045123    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 20:08:06.045012 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:06.044976    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 30 20:08:06.045197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:06.045041    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 30 20:08:07.050229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:07.050198    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:08:07.051396 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:07.051371    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:08:16.441009 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.440976    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:08:16.445546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.445521    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.499944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.499913    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:08:16.552443 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.549512    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-115a78-dockercfg-c9kj8\""
Oct 30 20:08:16.552443 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.550257    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-115a78\""
Oct 30 20:08:16.566349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.565930    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg6b4\" (UniqueName: \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.566349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.565983    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.585797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.585761    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:08:16.591231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.591203    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.649505 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.649474    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:08:16.667775 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.667744    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg6b4\" (UniqueName: \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.667946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.667787    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.668800 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.668774    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.726413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.726341    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg6b4\" (UniqueName: \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\") pod \"payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.759156 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.759130    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:16.768821 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.768797    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.769040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.769014    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts2p\" (UniqueName: \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.870788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.870755    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.870788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.870799    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts2p\" (UniqueName: \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.871845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.871818    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:16.930698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:16.930665    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts2p\" (UniqueName: \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\") pod \"payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:17.132017 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.131989    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:08:17.132558 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:17.132532    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43e8f75_e390_4387_ba59_6397bbabca5d.slice/crio-9769ad479ebb872008486cdfdd9b34f18bf2a0cec9759083dd5616d8a39e1ac7 WatchSource:0}: Error finding container 9769ad479ebb872008486cdfdd9b34f18bf2a0cec9759083dd5616d8a39e1ac7: Status 404 returned error can't find the container with id 9769ad479ebb872008486cdfdd9b34f18bf2a0cec9759083dd5616d8a39e1ac7
Oct 30 20:08:17.206416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.206391    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:17.578081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.578056    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:08:17.579890 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:17.579862    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7548cadb_6799_4cca_81f7_aac867ca467f.slice/crio-93cde4f3d77fe8ff88bef42ac208a798342252f21169abfab4f1ccfa70f1e3c4 WatchSource:0}: Error finding container 93cde4f3d77fe8ff88bef42ac208a798342252f21169abfab4f1ccfa70f1e3c4: Status 404 returned error can't find the container with id 93cde4f3d77fe8ff88bef42ac208a798342252f21169abfab4f1ccfa70f1e3c4
Oct 30 20:08:17.837896 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.837804    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" event={"ID":"7548cadb-6799-4cca-81f7-aac867ca467f","Type":"ContainerStarted","Data":"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"}
Oct 30 20:08:17.837896 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.837867    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" event={"ID":"7548cadb-6799-4cca-81f7-aac867ca467f","Type":"ContainerStarted","Data":"93cde4f3d77fe8ff88bef42ac208a798342252f21169abfab4f1ccfa70f1e3c4"}
Oct 30 20:08:17.838132 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.838007    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:17.839492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.839465    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" event={"ID":"e43e8f75-e390-4387-ba59-6397bbabca5d","Type":"ContainerStarted","Data":"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"}
Oct 30 20:08:17.839660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.839503    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" event={"ID":"e43e8f75-e390-4387-ba59-6397bbabca5d","Type":"ContainerStarted","Data":"9769ad479ebb872008486cdfdd9b34f18bf2a0cec9759083dd5616d8a39e1ac7"}
Oct 30 20:08:17.839660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.839624    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:17.951106 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:17.951058    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" podStartSLOduration=1.951043893 podStartE2EDuration="1.951043893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:17.949296011 +0000 UTC m=+2607.453699757" watchObservedRunningTime="2026-10-30 20:08:17.951043893 +0000 UTC m=+2607.455447647"
Oct 30 20:08:18.060929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:18.060880    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" podStartSLOduration=2.060865595 podStartE2EDuration="2.060865595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:18.059707448 +0000 UTC m=+2607.564111195" watchObservedRunningTime="2026-10-30 20:08:18.060865595 +0000 UTC m=+2607.565269369"
Oct 30 20:08:20.320198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.320164    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:08:20.320708 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.320384    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" podUID="82863459-0ecf-4be4-a316-d1fec173f4f6" containerName="maas-api" containerID="cri-o://13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089" gracePeriod=30
Oct 30 20:08:20.599628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.599602    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:08:20.613798 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.613768    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\") pod \"82863459-0ecf-4be4-a316-d1fec173f4f6\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") "
Oct 30 20:08:20.613955 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.613824    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\") pod \"82863459-0ecf-4be4-a316-d1fec173f4f6\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") "
Oct 30 20:08:20.613955 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.613855    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") pod \"82863459-0ecf-4be4-a316-d1fec173f4f6\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") "
Oct 30 20:08:20.613955 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.613930    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\" (UniqueName: \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\") pod \"82863459-0ecf-4be4-a316-d1fec173f4f6\" (UID: \"82863459-0ecf-4be4-a316-d1fec173f4f6\") "
Oct 30 20:08:20.614222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.614196    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca" pod "82863459-0ecf-4be4-a316-d1fec173f4f6" (UID: "82863459-0ecf-4be4-a316-d1fec173f4f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:20.614365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.614340    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82863459-0ecf-4be4-a316-d1fec173f4f6-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:20.616115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.616088    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs" pod "82863459-0ecf-4be4-a316-d1fec173f4f6" (UID: "82863459-0ecf-4be4-a316-d1fec173f4f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:20.616199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.616113    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls" pod "82863459-0ecf-4be4-a316-d1fec173f4f6" (UID: "82863459-0ecf-4be4-a316-d1fec173f4f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:20.616199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.616140    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx" pod "82863459-0ecf-4be4-a316-d1fec173f4f6" (UID: "82863459-0ecf-4be4-a316-d1fec173f4f6"). InnerVolumeSpecName "kube-api-access-xhbxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:20.716089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.716054    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbxx\" (UniqueName: \"kubernetes.io/projected/82863459-0ecf-4be4-a316-d1fec173f4f6-kube-api-access-xhbxx\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:20.716089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.716086    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:20.716398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.716099    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82863459-0ecf-4be4-a316-d1fec173f4f6-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:08:20.866622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.866519    5085 generic.go:356] "Generic (PLEG): container finished" podID="82863459-0ecf-4be4-a316-d1fec173f4f6" containerID="13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089" exitCode=0
Oct 30 20:08:20.866865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.866620    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"
Oct 30 20:08:20.866865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.866618    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" event={"ID":"82863459-0ecf-4be4-a316-d1fec173f4f6","Type":"ContainerDied","Data":"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"}
Oct 30 20:08:20.866865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.866689    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2" event={"ID":"82863459-0ecf-4be4-a316-d1fec173f4f6","Type":"ContainerDied","Data":"38836863d759ff11f0601a7f4bef603b122b04c65802e236edab3f23eabcc049"}
Oct 30 20:08:20.866865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.866721    5085 scope.go:122] "RemoveContainer" containerID="13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"
Oct 30 20:08:20.877245 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.877224    5085 scope.go:122] "RemoveContainer" containerID="13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"
Oct 30 20:08:20.877636 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:08:20.877601    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089\": container with ID starting with 13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089 not found: ID does not exist" containerID="13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"
Oct 30 20:08:20.877726 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:20.877648    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089"} err="failed to get container status \"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089\": rpc error: code = NotFound desc = could not find container \"13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089\": container with ID starting with 13e16d25b294cf064214c2e5a43c31f723f77ae24e1a7bd2010806a5a264d089 not found: ID does not exist"
Oct 30 20:08:21.024967 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:21.024934    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:08:21.065409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:21.065371    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f700fe-76b659669d-xrxr2"]
Oct 30 20:08:21.940536 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:08:21.940501    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:08:22.142293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:22.142254    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:08:22.633555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:22.633519    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82863459-0ecf-4be4-a316-d1fec173f4f6" path="/var/lib/kubelet/pods/82863459-0ecf-4be4-a316-d1fec173f4f6/volumes"
Oct 30 20:08:22.752461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:22.752428    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"]
Oct 30 20:08:22.757204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:22.757182    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f700fe-dmcpg"]
Oct 30 20:08:24.633029 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:24.632998    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186bbf5d-3506-4f3e-b43e-bb51dd01eb69" path="/var/lib/kubelet/pods/186bbf5d-3506-4f3e-b43e-bb51dd01eb69/volumes"
Oct 30 20:08:28.848245 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:28.848214    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:08:28.848690 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:28.848292    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:08:32.965480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.965446    5085 generic.go:356] "Generic (PLEG): container finished" podID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerID="71073c27b1578f5628d9c418cdc05777a49c978e42b14a9245ccbb41a195fc34" exitCode=0
Oct 30 20:08:32.965862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.965520    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerDied","Data":"71073c27b1578f5628d9c418cdc05777a49c978e42b14a9245ccbb41a195fc34"}
Oct 30 20:08:32.965965 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.965950    5085 scope.go:122] "RemoveContainer" containerID="71073c27b1578f5628d9c418cdc05777a49c978e42b14a9245ccbb41a195fc34"
Oct 30 20:08:32.967435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.967403    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:32.967503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.967475    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:32.967552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:32.967505    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:08:33.990202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:33.990166    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerStarted","Data":"5d26f2b309189b8cfa2f2733136ccd7dd21103a190549da1a3c49ccdc2860dc6"}
Oct 30 20:08:33.999649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:33.999623    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:08:34.984851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:34.984821    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:08:35.000693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:35.000661    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:08:35.001068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:35.000718    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 20:08:35.999621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:35.999576    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:08:35.999831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:35.999658    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 20:08:36.999737 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:36.999700    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:08:37.000228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:36.999756    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 20:08:38.000284 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:38.000244    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:08:38.000829 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:38.000310    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 20:08:39.000452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:39.000404    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Oct 30 20:08:39.000894 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:39.000510    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Oct 30 20:08:40.004490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:40.004454    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:08:40.005444 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:40.005426    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:08:41.431683 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.431641    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:08:41.437123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.437090    5085 cpu_manager.go:414] "containerMap: removing container" podUID="82863459-0ecf-4be4-a316-d1fec173f4f6" containerName="maas-api"
Oct 30 20:08:41.437123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.437123    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82863459-0ecf-4be4-a316-d1fec173f4f6" containerName="maas-api"
Oct 30 20:08:41.437349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.437327    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="82863459-0ecf-4be4-a316-d1fec173f4f6" containerName="maas-api"
Oct 30 20:08:41.442724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.442694    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.473298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.473265    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:08:41.493039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.493009    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-e6b14d\""
Oct 30 20:08:41.493748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.493728    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-e6b14d-dockercfg-bbz2v\""
Oct 30 20:08:41.550039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.550007    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:08:41.554370 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.554345    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.582979 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.582946    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.583165 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.583141    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpgg\" (UniqueName: \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.610716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.610684    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:08:41.684829 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.684741    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt55t\" (UniqueName: \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.684829 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.684801    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.685083 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.684886    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpgg\" (UniqueName: \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.685143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.685097    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.685790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.685769    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.743507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.743477    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpgg\" (UniqueName: \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\") pod \"payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.760413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.760390    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:41.786461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.786433    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt55t\" (UniqueName: \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.786584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.786497    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.787275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.787255    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.846165 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.846136    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt55t\" (UniqueName: \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\") pod \"payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:41.869062 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:41.869027    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:42.130662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:42.130527    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:08:42.131729 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:42.131697    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea45404_def4_49c2_b7d4_783608fb65cf.slice/crio-2740a3f707c81949b45d522369d01fcc330bf79e2270a11d3a52d6f46c87d19c WatchSource:0}: Error finding container 2740a3f707c81949b45d522369d01fcc330bf79e2270a11d3a52d6f46c87d19c: Status 404 returned error can't find the container with id 2740a3f707c81949b45d522369d01fcc330bf79e2270a11d3a52d6f46c87d19c
Oct 30 20:08:42.247786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:42.247764    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:08:42.248666 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:42.248638    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9093f0_44c2_4967_8279_19b55d3c102f.slice/crio-02299e20596c0eb7f03fc562f64926ee22c29c9c7a92714e86b041cb5c28c0d8 WatchSource:0}: Error finding container 02299e20596c0eb7f03fc562f64926ee22c29c9c7a92714e86b041cb5c28c0d8: Status 404 returned error can't find the container with id 02299e20596c0eb7f03fc562f64926ee22c29c9c7a92714e86b041cb5c28c0d8
Oct 30 20:08:43.053205 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.053160    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" event={"ID":"eea45404-def4-49c2-b7d4-783608fb65cf","Type":"ContainerStarted","Data":"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"}
Oct 30 20:08:43.053673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.053231    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:43.053673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.053260    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" event={"ID":"eea45404-def4-49c2-b7d4-783608fb65cf","Type":"ContainerStarted","Data":"2740a3f707c81949b45d522369d01fcc330bf79e2270a11d3a52d6f46c87d19c"}
Oct 30 20:08:43.055080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.055055    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" event={"ID":"0f9093f0-44c2-4967-8279-19b55d3c102f","Type":"ContainerStarted","Data":"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"}
Oct 30 20:08:43.055192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.055096    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" event={"ID":"0f9093f0-44c2-4967-8279-19b55d3c102f","Type":"ContainerStarted","Data":"02299e20596c0eb7f03fc562f64926ee22c29c9c7a92714e86b041cb5c28c0d8"}
Oct 30 20:08:43.055192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.055130    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:43.164425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.164374    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" podStartSLOduration=2.164362326 podStartE2EDuration="2.164362326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:43.164325702 +0000 UTC m=+2632.668729448" watchObservedRunningTime="2026-10-30 20:08:43.164362326 +0000 UTC m=+2632.668766072"
Oct 30 20:08:43.275286 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:43.275234    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" podStartSLOduration=2.275221185 podStartE2EDuration="2.275221185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:43.274753738 +0000 UTC m=+2632.779157487" watchObservedRunningTime="2026-10-30 20:08:43.275221185 +0000 UTC m=+2632.779624932"
Oct 30 20:08:51.940125 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:08:51.940089    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:08:54.065087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:54.065059    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:08:54.065534 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:54.065130    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:08:58.075343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.075265    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:08:58.081019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.080999    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.130187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.130147    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5chj4\" (UniqueName: \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.130346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.130326    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.133292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.133274    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a3c171\""
Oct 30 20:08:58.180509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.180475    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a3c171-dockercfg-d2cjk\""
Oct 30 20:08:58.184439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.184408    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:08:58.204643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.204609    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:08:58.217712 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.217682    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.232054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.232021    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5chj4\" (UniqueName: \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.232252 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.232195    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.233194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.233166    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.257052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.257022    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:08:58.292128 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.292098    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5chj4\" (UniqueName: \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\") pod \"payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.333226 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.333147    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrw8\" (UniqueName: \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.333397 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.333384    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.394321 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.394294    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:58.434960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.434928    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrw8\" (UniqueName: \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.435149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.435128    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.435831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.435812    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.535284 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.535254    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrw8\" (UniqueName: \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\") pod \"payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.536273 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.535846    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:58.766463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.766433    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:08:58.767622 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:58.767598    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05645f4f_d82a_401a_bff5_480210fe4062.slice/crio-8041d8a944d041929412b832659a8133def2c549af0a0b955060843ac7ef42ff WatchSource:0}: Error finding container 8041d8a944d041929412b832659a8133def2c549af0a0b955060843ac7ef42ff: Status 404 returned error can't find the container with id 8041d8a944d041929412b832659a8133def2c549af0a0b955060843ac7ef42ff
Oct 30 20:08:58.915078 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:58.915039    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:08:58.916261 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:08:58.916235    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5cf2cb_ef10_4542_8b21_d4efcdf8b71e.slice/crio-9c4d6c1bedc52b6d19a6eb6bcc17be3f95455bf34b67139191fcd2f1b531fd83 WatchSource:0}: Error finding container 9c4d6c1bedc52b6d19a6eb6bcc17be3f95455bf34b67139191fcd2f1b531fd83: Status 404 returned error can't find the container with id 9c4d6c1bedc52b6d19a6eb6bcc17be3f95455bf34b67139191fcd2f1b531fd83
Oct 30 20:08:59.187407 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.187373    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" event={"ID":"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e","Type":"ContainerStarted","Data":"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"}
Oct 30 20:08:59.187891 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.187426    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" event={"ID":"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e","Type":"ContainerStarted","Data":"9c4d6c1bedc52b6d19a6eb6bcc17be3f95455bf34b67139191fcd2f1b531fd83"}
Oct 30 20:08:59.187891 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.187589    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:08:59.189057 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.189030    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" event={"ID":"05645f4f-d82a-401a-bff5-480210fe4062","Type":"ContainerStarted","Data":"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"}
Oct 30 20:08:59.189195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.189070    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" event={"ID":"05645f4f-d82a-401a-bff5-480210fe4062","Type":"ContainerStarted","Data":"8041d8a944d041929412b832659a8133def2c549af0a0b955060843ac7ef42ff"}
Oct 30 20:08:59.189195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.189141    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:08:59.300040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.299988    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" podStartSLOduration=1.299974544 podStartE2EDuration="1.299974544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:59.297503578 +0000 UTC m=+2648.801907367" watchObservedRunningTime="2026-10-30 20:08:59.299974544 +0000 UTC m=+2648.804378291"
Oct 30 20:08:59.409766 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:08:59.409717    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" podStartSLOduration=1.409702405 podStartE2EDuration="1.409702405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:59.407286559 +0000 UTC m=+2648.911690306" watchObservedRunningTime="2026-10-30 20:08:59.409702405 +0000 UTC m=+2648.914106151"
Oct 30 20:09:04.240042 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.240009    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:09:04.244662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.244634    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.306681 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.306644    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqhjm\" (UniqueName: \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.306869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.306708    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.342169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.342138    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f53eb9-dockercfg-vpp65\""
Oct 30 20:09:04.342366 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.342198    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f53eb9\""
Oct 30 20:09:04.346844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.346819    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:09:04.408297 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.408254    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.408546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.408526    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqhjm\" (UniqueName: \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.409998 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.409966    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.418465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.418433    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:09:04.423049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.423030    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.448903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.448873    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:09:04.467873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.467845    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqhjm\" (UniqueName: \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\") pod \"payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.509802 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.509737    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.509942 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.509876    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbb5\" (UniqueName: \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.559574 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.559532    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:04.611092 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.611052    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.611292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.611184    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbb5\" (UniqueName: \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.611925 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.611898    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.676125 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.676092    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbb5\" (UniqueName: \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\") pod \"payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.737713 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.737684    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:04.938602 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:04.938558    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:09:04.939714 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:09:04.939675    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf996f1e3_ae0e_4f90_9e93_86ce7e747db0.slice/crio-9899e6bc4e3ee3271a7e00b6240e85fc4b0bb2bbe6740fa72e781f4d675e2af2 WatchSource:0}: Error finding container 9899e6bc4e3ee3271a7e00b6240e85fc4b0bb2bbe6740fa72e781f4d675e2af2: Status 404 returned error can't find the container with id 9899e6bc4e3ee3271a7e00b6240e85fc4b0bb2bbe6740fa72e781f4d675e2af2
Oct 30 20:09:05.110570 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.110536    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:09:05.111962 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:09:05.111929    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5c323e_bfc2_47a8_85fc_ef9a96537aaa.slice/crio-07768000bd5c922b56b47d07f0957fdd9d718f9c60bcf701152db7000a3921ff WatchSource:0}: Error finding container 07768000bd5c922b56b47d07f0957fdd9d718f9c60bcf701152db7000a3921ff: Status 404 returned error can't find the container with id 07768000bd5c922b56b47d07f0957fdd9d718f9c60bcf701152db7000a3921ff
Oct 30 20:09:05.232004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.231975    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" event={"ID":"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa","Type":"ContainerStarted","Data":"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"}
Oct 30 20:09:05.232172 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.232025    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" event={"ID":"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa","Type":"ContainerStarted","Data":"07768000bd5c922b56b47d07f0957fdd9d718f9c60bcf701152db7000a3921ff"}
Oct 30 20:09:05.232234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.232174    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:05.233750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.233725    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" event={"ID":"f996f1e3-ae0e-4f90-9e93-86ce7e747db0","Type":"ContainerStarted","Data":"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"}
Oct 30 20:09:05.233870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.233772    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" event={"ID":"f996f1e3-ae0e-4f90-9e93-86ce7e747db0","Type":"ContainerStarted","Data":"9899e6bc4e3ee3271a7e00b6240e85fc4b0bb2bbe6740fa72e781f4d675e2af2"}
Oct 30 20:09:05.233870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.233858    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:05.345113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.345066    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" podStartSLOduration=1.345052353 podStartE2EDuration="1.345052353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:05.341702045 +0000 UTC m=+2654.846105792" watchObservedRunningTime="2026-10-30 20:09:05.345052353 +0000 UTC m=+2654.849456115"
Oct 30 20:09:05.453470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:05.453411    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" podStartSLOduration=1.453397694 podStartE2EDuration="1.453397694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:05.451156761 +0000 UTC m=+2654.955560524" watchObservedRunningTime="2026-10-30 20:09:05.453397694 +0000 UTC m=+2654.957801440"
Oct 30 20:09:09.457189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:09.457144    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:09:10.198935 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:10.198903    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:09:10.199110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:10.199066    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:09:16.243235 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:16.243207    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:09:16.243667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:16.243285    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:09:20.111354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.111322    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:09:20.116240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.116218    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.165989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.165962    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-115a78-serving-cert\""
Oct 30 20:09:20.199865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.199828    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ftf\" (UniqueName: \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.200088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.199906    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.200088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.199974    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.200088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.200055    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.216155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.216127    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:09:20.301554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.301519    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ftf\" (UniqueName: \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.301767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.301595    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.301767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.301616    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.301767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.301641    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.301767 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:20.301731    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-serving-cert: secret "maas-api-e2e-praxis-b-115a78-serving-cert" not found
Oct 30 20:09:20.302018 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:20.301797    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls podName:33f839db-30a0-4ec2-ac02-e477a104e511 nodeName:}" failed. No retries permitted until 2026-10-30 20:09:20.801779105 +0000 UTC m=+2670.306182844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls") pod "maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" (UID: "33f839db-30a0-4ec2-ac02-e477a104e511") : secret "maas-api-e2e-praxis-b-115a78-serving-cert" not found
Oct 30 20:09:20.302280 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.302261    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.304120 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.304100    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.362258 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.362173    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ftf\" (UniqueName: \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.807503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.807460    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:20.809886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:20.809859    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") pod \"maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:21.030906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:21.030875    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:21.405688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:21.405665    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:09:21.407375 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:09:21.407347    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f839db_30a0_4ec2_ac02_e477a104e511.slice/crio-39a66ff571df207d5856a7c5d4aa1d001afccb3b26ee1ea8dcb72faea1a05cee WatchSource:0}: Error finding container 39a66ff571df207d5856a7c5d4aa1d001afccb3b26ee1ea8dcb72faea1a05cee: Status 404 returned error can't find the container with id 39a66ff571df207d5856a7c5d4aa1d001afccb3b26ee1ea8dcb72faea1a05cee
Oct 30 20:09:21.940262 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:21.940171    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:09:22.361435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:22.361400    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" event={"ID":"33f839db-30a0-4ec2-ac02-e477a104e511","Type":"ContainerStarted","Data":"39a66ff571df207d5856a7c5d4aa1d001afccb3b26ee1ea8dcb72faea1a05cee"}
Oct 30 20:09:23.370949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:23.370910    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" event={"ID":"33f839db-30a0-4ec2-ac02-e477a104e511","Type":"ContainerStarted","Data":"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"}
Oct 30 20:09:23.371497 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:23.371008    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:23.487288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:23.487234    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" podStartSLOduration=2.570143356 podStartE2EDuration="3.487219513s" totalImagesPullingTime="917.076157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:23.484435151 +0000 UTC m=+2672.988838923" watchObservedRunningTime="2026-10-30 20:09:23.487219513 +0000 UTC m=+2672.991623314"
Oct 30 20:09:29.382957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:29.382930    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:09:45.320203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.320159    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:09:45.324908 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.324885    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.411159 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.411123    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.411325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.411203    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.411325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.411317    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cv4d\" (UniqueName: \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.411417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.411355    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.423102 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.423078    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-e6b14d-serving-cert\""
Oct 30 20:09:45.426661 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.426639    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:09:45.512637 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.512607    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cv4d\" (UniqueName: \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.512839 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.512647    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.512907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.512835    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.512907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.512895    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.513060 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:45.513040    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-serving-cert: secret "maas-api-e2e-praxis-a-e6b14d-serving-cert" not found
Oct 30 20:09:45.513116 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:45.513104    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls podName:bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6 nodeName:}" failed. No retries permitted until 2026-10-30 20:09:46.013085845 +0000 UTC m=+2695.517489574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls") pod "maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" (UID: "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6") : secret "maas-api-e2e-praxis-a-e6b14d-serving-cert" not found
Oct 30 20:09:45.513492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.513472    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.515328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.515301    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:45.572003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:45.571927    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cv4d\" (UniqueName: \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:46.020257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:46.020224    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:46.022745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:46.022712    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:46.241057 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:46.241023    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:46.623535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:46.623506    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:09:46.625678 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:09:46.625648    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5b5b5d_39a1_4de6_af96_2c3fd5fb5ad6.slice/crio-c2b26556e06aa701505a7d976d68296da52f1af64efb429ea18523331b511dcf WatchSource:0}: Error finding container c2b26556e06aa701505a7d976d68296da52f1af64efb429ea18523331b511dcf: Status 404 returned error can't find the container with id c2b26556e06aa701505a7d976d68296da52f1af64efb429ea18523331b511dcf
Oct 30 20:09:47.552402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:47.552363    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" event={"ID":"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6","Type":"ContainerStarted","Data":"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"}
Oct 30 20:09:47.552579 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:47.552463    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:09:47.552579 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:47.552486    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" event={"ID":"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6","Type":"ContainerStarted","Data":"c2b26556e06aa701505a7d976d68296da52f1af64efb429ea18523331b511dcf"}
Oct 30 20:09:47.694826 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:47.694763    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" podStartSLOduration=1.859450322 podStartE2EDuration="2.694749265s" totalImagesPullingTime="835.298943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:47.691868403 +0000 UTC m=+2697.196272149" watchObservedRunningTime="2026-10-30 20:09:47.694749265 +0000 UTC m=+2697.199153013"
Oct 30 20:09:51.298774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:51.298741    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:09:51.304168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:51.304143    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:09:51.940366 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:09:51.940329    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:09:53.566538 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:09:53.566511    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:10:07.485019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.484987    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:10:07.490891 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.490870    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.542291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.542253    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a3c171-serving-cert\""
Oct 30 20:10:07.580002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.579960    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j849q\" (UniqueName: \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.580216 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.580187    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.580296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.580245    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.580399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.580376    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.592415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.592389    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:10:07.682200 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.682166    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.682409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.682208    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.682409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.682262    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.682409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.682370    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j849q\" (UniqueName: \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.682635 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:07.682464    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-serving-cert: secret "maas-api-e2e-praxis-b-a3c171-serving-cert" not found
Oct 30 20:10:07.682635 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:07.682591    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls podName:40dc596b-34f4-4871-83fc-066df0d09776 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:08.182539055 +0000 UTC m=+2717.686942800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls") pod "maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" (UID: "40dc596b-34f4-4871-83fc-066df0d09776") : secret "maas-api-e2e-praxis-b-a3c171-serving-cert" not found
Oct 30 20:10:07.683057 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.683038    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.684833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.684810    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:07.744178 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:07.744098    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j849q\" (UniqueName: \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:08.189354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:08.189313    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:08.191835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:08.191812    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:08.405201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:08.405168    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:08.805373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:08.805344    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:10:08.807933 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:10:08.807910    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dc596b_34f4_4871_83fc_066df0d09776.slice/crio-42b84c3a17a33588e1c79310a86ee8d8579811bf7c3dd6d72e710c0cb6c0caed WatchSource:0}: Error finding container 42b84c3a17a33588e1c79310a86ee8d8579811bf7c3dd6d72e710c0cb6c0caed: Status 404 returned error can't find the container with id 42b84c3a17a33588e1c79310a86ee8d8579811bf7c3dd6d72e710c0cb6c0caed
Oct 30 20:10:08.809745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:08.809729    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:10:09.668177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:09.668143    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:10:09.728711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:09.727827    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" event={"ID":"40dc596b-34f4-4871-83fc-066df0d09776","Type":"ContainerStarted","Data":"42b84c3a17a33588e1c79310a86ee8d8579811bf7c3dd6d72e710c0cb6c0caed"}
Oct 30 20:10:10.736578 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:10.736527    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" event={"ID":"40dc596b-34f4-4871-83fc-066df0d09776","Type":"ContainerStarted","Data":"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"}
Oct 30 20:10:10.737003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:10.736637    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:10.852179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:10.852127    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" podStartSLOduration=2.714852325 podStartE2EDuration="3.852112236s" totalImagesPullingTime="1.137259911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:10.84870449 +0000 UTC m=+2720.353108236" watchObservedRunningTime="2026-10-30 20:10:10.852112236 +0000 UTC m=+2720.356515982"
Oct 30 20:10:16.749615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:16.749585    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:10:17.833268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.833235    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:10:17.837882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.837863    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:17.911509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.911475    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:17.911509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.911515    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:17.911815 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.911551    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:17.911815 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.911650    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:17.936310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.936277    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f53eb9-serving-cert\""
Oct 30 20:10:17.940053 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:17.940028    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:10:18.012934 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.012904    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.013104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.012941    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.013104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.013051    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.013232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.013112    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.013232 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:18.013215    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-serving-cert: secret "maas-api-e2e-praxis-b-f53eb9-serving-cert" not found
Oct 30 20:10:18.013351 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:18.013288    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls podName:ea7c11c2-93ad-4a3d-8960-0f39aa993bd2 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:18.513263341 +0000 UTC m=+2728.017667080 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls") pod "maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" (UID: "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2") : secret "maas-api-e2e-praxis-b-f53eb9-serving-cert" not found
Oct 30 20:10:18.013872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.013849    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.015603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.015580    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.072506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.072475    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.521502 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.521455    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.524346 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.524318    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:18.752516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:18.752487    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:19.130047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:19.130016    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:10:19.130467 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:10:19.130204    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7c11c2_93ad_4a3d_8960_0f39aa993bd2.slice/crio-d8fd5f8627b95a821cdc641c50d915a6be95d014da3ca83efde5a0cae9e7acc8 WatchSource:0}: Error finding container d8fd5f8627b95a821cdc641c50d915a6be95d014da3ca83efde5a0cae9e7acc8: Status 404 returned error can't find the container with id d8fd5f8627b95a821cdc641c50d915a6be95d014da3ca83efde5a0cae9e7acc8
Oct 30 20:10:19.525989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:19.525957    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:10:19.777226 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:19.777144    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:10:19.803124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:19.803080    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" event={"ID":"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2","Type":"ContainerStarted","Data":"d8fd5f8627b95a821cdc641c50d915a6be95d014da3ca83efde5a0cae9e7acc8"}
Oct 30 20:10:20.811673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:20.811633    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" event={"ID":"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2","Type":"ContainerStarted","Data":"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"}
Oct 30 20:10:20.812076 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:20.811783    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:20.926957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:20.926903    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" podStartSLOduration=3.053945092 podStartE2EDuration="3.9268884s" totalImagesPullingTime="872.943308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:20.924889967 +0000 UTC m=+2730.429293714" watchObservedRunningTime="2026-10-30 20:10:20.9268884 +0000 UTC m=+2730.431292150"
Oct 30 20:10:21.938879 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:21.938829    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:10:25.824311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:25.824264    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 20:10:25.824786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:25.824498    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" containerID="cri-o://42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e" gracePeriod=30
Oct 30 20:10:26.825107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:26.825077    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:10:30.130011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:30.129919    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 20:10:30.130476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:30.130206    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" containerID="cri-o://75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde" gracePeriod=30
Oct 30 20:10:30.402676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:30.402599    5085 patch_prober.go:29] interesting pod/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:30.402676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:30.402660    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:31.093216 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.093161    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 20:10:31.179079 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179049    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179105    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179171    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179246    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179298    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179367    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179469    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179535    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179599    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179615    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\") pod \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\" (UID: \"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f\") "
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179634    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179649    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.179874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.179826    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.180182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180017    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.180263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180239    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.180326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180266    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.180326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180281    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.180326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180295    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.180326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.180309    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.181656 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.181624    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:31.181656 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.181636    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:31.181888 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.181868    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "kube-api-access-dbgdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:31.181999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.181894    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token" pod "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" (UID: "45bb3c5d-f95c-47bc-990d-5a91e35d3d2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:31.282319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.282278    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.282319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.282318    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbgdf\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-kube-api-access-dbgdf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.282581 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.282334    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.282581 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.282348    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:31.897619 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.897587    5085 generic.go:356] "Generic (PLEG): container finished" podID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerID="42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e" exitCode=0
Oct 30 20:10:31.897832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.897623    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"
Oct 30 20:10:31.897832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.897652    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerDied","Data":"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"}
Oct 30 20:10:31.897832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.897705    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc" event={"ID":"45bb3c5d-f95c-47bc-990d-5a91e35d3d2f","Type":"ContainerDied","Data":"1a120d14265f006729d7de9ce4f460300274127531e39bacb1fc56d9749c7b79"}
Oct 30 20:10:31.897832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.897726    5085 scope.go:122] "RemoveContainer" containerID="42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"
Oct 30 20:10:31.909207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.909190    5085 scope.go:122] "RemoveContainer" containerID="3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"
Oct 30 20:10:31.918923 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.918903    5085 scope.go:122] "RemoveContainer" containerID="42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"
Oct 30 20:10:31.919173 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:31.919153    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e\": container with ID starting with 42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e not found: ID does not exist" containerID="42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"
Oct 30 20:10:31.919254 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.919181    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e"} err="failed to get container status \"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e\": rpc error: code = NotFound desc = could not find container \"42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e\": container with ID starting with 42a6877190a0e4758f52b61af86f3ffb34593aab723c17d988d3085a2319b74e not found: ID does not exist"
Oct 30 20:10:31.919254 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.919202    5085 scope.go:122] "RemoveContainer" containerID="3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"
Oct 30 20:10:31.919417 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:31.919402    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b\": container with ID starting with 3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b not found: ID does not exist" containerID="3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"
Oct 30 20:10:31.919461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:31.919421    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b"} err="failed to get container status \"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b\": rpc error: code = NotFound desc = could not find container \"3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b\": container with ID starting with 3305ea2ac1f3da280373ce68b2ce799290f9e118fc709710566f7d40f2427e6b not found: ID does not exist"
Oct 30 20:10:32.103398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:32.103370    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 20:10:32.155245 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:32.155152    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ec609-openshift-default-66fc5976ff-k7lgc"]
Oct 30 20:10:32.413548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:32.413474    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:10:32.413938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:32.413543    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:10:32.632338 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:32.632303    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" path="/var/lib/kubelet/pods/45bb3c5d-f95c-47bc-990d-5a91e35d3d2f/volumes"
Oct 30 20:10:35.414105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.414042    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 20:10:35.537593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537546    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537610    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537638    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537661    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537687    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537959 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537820    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.537959 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537844    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:35.537959 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537933    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.538118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.537989    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:35.538118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538018    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.538118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538026    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:35.538118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538060    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\") pod \"1ea11f0c-7bc5-4c90-8882-df523acac165\" (UID: \"1ea11f0c-7bc5-4c90-8882-df523acac165\") "
Oct 30 20:10:35.538118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538095    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:35.538414 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538373    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:35.538624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538606    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.538704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538628    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea11f0c-7bc5-4c90-8882-df523acac165-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.538704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538642    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.538704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538657    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.538704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.538670    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.540086 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.540059    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "kube-api-access-bcrgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:35.540185 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.540091    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:35.540270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.540253    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:10:35.540325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.540287    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo" pod "1ea11f0c-7bc5-4c90-8882-df523acac165" (UID: "1ea11f0c-7bc5-4c90-8882-df523acac165"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:10:35.639526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.639492    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.639526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.639519    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.639526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.639529    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea11f0c-7bc5-4c90-8882-df523acac165-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.639799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.639538    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrgp\" (UniqueName: \"kubernetes.io/projected/1ea11f0c-7bc5-4c90-8882-df523acac165-kube-api-access-bcrgp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:10:35.930049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.930013    5085 generic.go:356] "Generic (PLEG): container finished" podID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerID="75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde" exitCode=0
Oct 30 20:10:35.930266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.930075    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"
Oct 30 20:10:35.930266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.930101    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerDied","Data":"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"}
Oct 30 20:10:35.930266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.930170    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z" event={"ID":"1ea11f0c-7bc5-4c90-8882-df523acac165","Type":"ContainerDied","Data":"8e22d8f8664a69c0331d723291f01d0ac2ab448d01da3c1fb59694bf4a926071"}
Oct 30 20:10:35.930266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.930200    5085 scope.go:122] "RemoveContainer" containerID="75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"
Oct 30 20:10:35.941239 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.941222    5085 scope.go:122] "RemoveContainer" containerID="df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"
Oct 30 20:10:35.950683 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.950658    5085 scope.go:122] "RemoveContainer" containerID="75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"
Oct 30 20:10:35.950964 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:35.950942    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde\": container with ID starting with 75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde not found: ID does not exist" containerID="75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"
Oct 30 20:10:35.951031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.950972    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde"} err="failed to get container status \"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde\": rpc error: code = NotFound desc = could not find container \"75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde\": container with ID starting with 75f1c8d219c53cb1af8355b7978e3c1613f84c45ab6ab6d1cc5029a59e325dde not found: ID does not exist"
Oct 30 20:10:35.951031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.950990    5085 scope.go:122] "RemoveContainer" containerID="df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"
Oct 30 20:10:35.951232 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:35.951213    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa\": container with ID starting with df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa not found: ID does not exist" containerID="df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"
Oct 30 20:10:35.951286 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:35.951244    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa"} err="failed to get container status \"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa\": rpc error: code = NotFound desc = could not find container \"df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa\": container with ID starting with df864d80ee06ced9048b3129b2b64d6d224a2ca67d5d13c54a50eedf0be743fa not found: ID does not exist"
Oct 30 20:10:36.096025 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:36.095994    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 20:10:36.142719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:36.142687    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-9q89z"]
Oct 30 20:10:36.632812 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:10:36.632773    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" path="/var/lib/kubelet/pods/1ea11f0c-7bc5-4c90-8882-df523acac165/volumes"
Oct 30 20:10:51.940007 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:10:51.939968    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:11:04.282550 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:04.282514    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:11:04.498054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:04.498020    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:11:04.576777 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:04.576701    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:11:09.498665 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.498626    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" podUID="a091513f-11f1-4cfe-8d1a-7fc257726f91" containerName="payload-processing" containerID="cri-o://9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de" gracePeriod=25
Oct 30 20:11:09.577704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.577669    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" podUID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" containerName="payload-pre-processing" containerID="cri-o://70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8" gracePeriod=25
Oct 30 20:11:09.856350 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.856319    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:11:09.885547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.885524    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:11:09.947405 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947374    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\") pod \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") "
Oct 30 20:11:09.947597 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947576    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\") pod \"a091513f-11f1-4cfe-8d1a-7fc257726f91\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") "
Oct 30 20:11:09.947677 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947620    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\" (UniqueName: \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\") pod \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\" (UID: \"1cfdcc42-daef-46b2-af7b-66527e8bcaa1\") "
Oct 30 20:11:09.947677 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947656    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\" (UniqueName: \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\") pod \"a091513f-11f1-4cfe-8d1a-7fc257726f91\" (UID: \"a091513f-11f1-4cfe-8d1a-7fc257726f91\") "
Oct 30 20:11:09.947837 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947815    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume" pod "1cfdcc42-daef-46b2-af7b-66527e8bcaa1" (UID: "1cfdcc42-daef-46b2-af7b-66527e8bcaa1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:09.947899 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.947877    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume" pod "a091513f-11f1-4cfe-8d1a-7fc257726f91" (UID: "a091513f-11f1-4cfe-8d1a-7fc257726f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:09.948094 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.948068    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a091513f-11f1-4cfe-8d1a-7fc257726f91-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:09.948094 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.948087    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:09.949717 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.949690    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb" pod "1cfdcc42-daef-46b2-af7b-66527e8bcaa1" (UID: "1cfdcc42-daef-46b2-af7b-66527e8bcaa1"). InnerVolumeSpecName "kube-api-access-6zxhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:09.949810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:09.949757    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l" pod "a091513f-11f1-4cfe-8d1a-7fc257726f91" (UID: "a091513f-11f1-4cfe-8d1a-7fc257726f91"). InnerVolumeSpecName "kube-api-access-x2n6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:10.049898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.049807    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxhb\" (UniqueName: \"kubernetes.io/projected/1cfdcc42-daef-46b2-af7b-66527e8bcaa1-kube-api-access-6zxhb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:10.049898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.049838    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2n6l\" (UniqueName: \"kubernetes.io/projected/a091513f-11f1-4cfe-8d1a-7fc257726f91-kube-api-access-x2n6l\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:10.178777 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.178740    5085 generic.go:356] "Generic (PLEG): container finished" podID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" containerID="70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8" exitCode=0
Oct 30 20:11:10.178963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.178788    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"
Oct 30 20:11:10.178963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.178823    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" event={"ID":"1cfdcc42-daef-46b2-af7b-66527e8bcaa1","Type":"ContainerDied","Data":"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"}
Oct 30 20:11:10.178963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.178888    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2" event={"ID":"1cfdcc42-daef-46b2-af7b-66527e8bcaa1","Type":"ContainerDied","Data":"38528221e372f7b788182afc5267abb7e0d28583df7ed238f05e96f3bb130fa6"}
Oct 30 20:11:10.178963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.178918    5085 scope.go:122] "RemoveContainer" containerID="70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"
Oct 30 20:11:10.180586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.180532    5085 generic.go:356] "Generic (PLEG): container finished" podID="a091513f-11f1-4cfe-8d1a-7fc257726f91" containerID="9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de" exitCode=0
Oct 30 20:11:10.180718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.180602    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"
Oct 30 20:11:10.180718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.180604    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" event={"ID":"a091513f-11f1-4cfe-8d1a-7fc257726f91","Type":"ContainerDied","Data":"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"}
Oct 30 20:11:10.180718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.180664    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6" event={"ID":"a091513f-11f1-4cfe-8d1a-7fc257726f91","Type":"ContainerDied","Data":"880a6443b630dd63ffe76dcaa9198c381151a80bd07e1306aedd9ed11571a84b"}
Oct 30 20:11:10.190480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.190464    5085 scope.go:122] "RemoveContainer" containerID="70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"
Oct 30 20:11:10.190918 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:10.190766    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8\": container with ID starting with 70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8 not found: ID does not exist" containerID="70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"
Oct 30 20:11:10.190918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.190789    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8"} err="failed to get container status \"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8\": rpc error: code = NotFound desc = could not find container \"70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8\": container with ID starting with 70867b2fe3c65174dbf9ef496228bcdd59ef940ab5926b66e9bf2599b0b5ada8 not found: ID does not exist"
Oct 30 20:11:10.190918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.190807    5085 scope.go:122] "RemoveContainer" containerID="9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"
Oct 30 20:11:10.200121 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.200105    5085 scope.go:122] "RemoveContainer" containerID="9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"
Oct 30 20:11:10.200383 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:10.200362    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de\": container with ID starting with 9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de not found: ID does not exist" containerID="9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"
Oct 30 20:11:10.200451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.200391    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de"} err="failed to get container status \"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de\": rpc error: code = NotFound desc = could not find container \"9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de\": container with ID starting with 9889139f29643884776b80bcc3b8a958c7b0a785299c73c3b74927fba516b0de not found: ID does not exist"
Oct 30 20:11:10.336790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.336709    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:11:10.342543 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.342509    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f61bae-795969f54c-zrwc2"]
Oct 30 20:11:10.447123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.447093    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:11:10.453301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.453274    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f61bae-86bcc7b558-4khb6"]
Oct 30 20:11:10.632597 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.632494    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" path="/var/lib/kubelet/pods/1cfdcc42-daef-46b2-af7b-66527e8bcaa1/volumes"
Oct 30 20:11:10.632993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:10.632940    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a091513f-11f1-4cfe-8d1a-7fc257726f91" path="/var/lib/kubelet/pods/a091513f-11f1-4cfe-8d1a-7fc257726f91/volumes"
Oct 30 20:11:21.940621 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:21.940551    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:11:40.293383 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.293344    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:11:40.293963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.293537    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" podUID="995bc9bc-93f8-44eb-840a-70f3a133ce88" containerName="maas-api" containerID="cri-o://d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e" gracePeriod=30
Oct 30 20:11:40.581345 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.581316    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:11:40.654513 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.654479    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\") pod \"995bc9bc-93f8-44eb-840a-70f3a133ce88\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") "
Oct 30 20:11:40.654735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.654545    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") pod \"995bc9bc-93f8-44eb-840a-70f3a133ce88\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") "
Oct 30 20:11:40.654735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.654674    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\" (UniqueName: \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\") pod \"995bc9bc-93f8-44eb-840a-70f3a133ce88\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") "
Oct 30 20:11:40.654735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.654724    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\") pod \"995bc9bc-93f8-44eb-840a-70f3a133ce88\" (UID: \"995bc9bc-93f8-44eb-840a-70f3a133ce88\") "
Oct 30 20:11:40.655159 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.655126    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca" pod "995bc9bc-93f8-44eb-840a-70f3a133ce88" (UID: "995bc9bc-93f8-44eb-840a-70f3a133ce88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:40.657222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.657194    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j" pod "995bc9bc-93f8-44eb-840a-70f3a133ce88" (UID: "995bc9bc-93f8-44eb-840a-70f3a133ce88"). InnerVolumeSpecName "kube-api-access-s8c2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:40.657365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.657252    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs" pod "995bc9bc-93f8-44eb-840a-70f3a133ce88" (UID: "995bc9bc-93f8-44eb-840a-70f3a133ce88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:40.657365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.657273    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls" pod "995bc9bc-93f8-44eb-840a-70f3a133ce88" (UID: "995bc9bc-93f8-44eb-840a-70f3a133ce88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:40.757189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.756971    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.757189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.757003    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/995bc9bc-93f8-44eb-840a-70f3a133ce88-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.757189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.757020    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8c2j\" (UniqueName: \"kubernetes.io/projected/995bc9bc-93f8-44eb-840a-70f3a133ce88-kube-api-access-s8c2j\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:40.757189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:40.757039    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/995bc9bc-93f8-44eb-840a-70f3a133ce88-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:41.404338 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.404303    5085 generic.go:356] "Generic (PLEG): container finished" podID="995bc9bc-93f8-44eb-840a-70f3a133ce88" containerID="d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e" exitCode=0
Oct 30 20:11:41.404850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.404365    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"
Oct 30 20:11:41.404850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.404385    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" event={"ID":"995bc9bc-93f8-44eb-840a-70f3a133ce88","Type":"ContainerDied","Data":"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"}
Oct 30 20:11:41.404850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.404470    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb" event={"ID":"995bc9bc-93f8-44eb-840a-70f3a133ce88","Type":"ContainerDied","Data":"509f19c39b0a809a608a3d079bda0403280382bc0ca8556a8aa28af0540f512e"}
Oct 30 20:11:41.404850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.404492    5085 scope.go:122] "RemoveContainer" containerID="d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"
Oct 30 20:11:41.416675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.416654    5085 scope.go:122] "RemoveContainer" containerID="d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"
Oct 30 20:11:41.416982 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:41.416957    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e\": container with ID starting with d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e not found: ID does not exist" containerID="d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"
Oct 30 20:11:41.417102 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.416993    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e"} err="failed to get container status \"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e\": rpc error: code = NotFound desc = could not find container \"d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e\": container with ID starting with d83555da9a42242d1531fd7367d5376d049c85d3d9b4301cfd60a4da5e90a39e not found: ID does not exist"
Oct 30 20:11:41.566136 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.566100    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:11:41.610832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:41.610800    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f61bae-bfc59cc78-bbnxb"]
Oct 30 20:11:42.633075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:42.633044    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="995bc9bc-93f8-44eb-840a-70f3a133ce88" path="/var/lib/kubelet/pods/995bc9bc-93f8-44eb-840a-70f3a133ce88/volumes"
Oct 30 20:11:47.725670 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:47.725627    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:11:47.837264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:47.837231    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:11:51.940189 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:51.940151    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:11:52.726819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:52.726767    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" podUID="d6cd9070-9c4b-4d84-8fde-96701967ea36" containerName="payload-processing" containerID="cri-o://3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad" gracePeriod=25
Oct 30 20:11:52.838272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:52.838238    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" podUID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" containerName="payload-pre-processing" containerID="cri-o://8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87" gracePeriod=25
Oct 30 20:11:52.995313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:52.995235    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:11:53.101652 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.101626    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\" (UniqueName: \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\") pod \"d6cd9070-9c4b-4d84-8fde-96701967ea36\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") "
Oct 30 20:11:53.101787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.101713    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\") pod \"d6cd9070-9c4b-4d84-8fde-96701967ea36\" (UID: \"d6cd9070-9c4b-4d84-8fde-96701967ea36\") "
Oct 30 20:11:53.102060 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.102039    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume" pod "d6cd9070-9c4b-4d84-8fde-96701967ea36" (UID: "d6cd9070-9c4b-4d84-8fde-96701967ea36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:53.103607 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.103585    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn" pod "d6cd9070-9c4b-4d84-8fde-96701967ea36" (UID: "d6cd9070-9c4b-4d84-8fde-96701967ea36"). InnerVolumeSpecName "kube-api-access-87mtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:53.105002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.104984    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:11:53.202665 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.202636    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\") pod \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") "
Oct 30 20:11:53.202851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.202679    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\" (UniqueName: \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\") pod \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\" (UID: \"9f08ea7a-a4f7-41bc-a450-636a96ffa72c\") "
Oct 30 20:11:53.203029 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.203008    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume" pod "9f08ea7a-a4f7-41bc-a450-636a96ffa72c" (UID: "9f08ea7a-a4f7-41bc-a450-636a96ffa72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:53.203153 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.203139    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6cd9070-9c4b-4d84-8fde-96701967ea36-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:53.203189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.203155    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:53.203189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.203165    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mtn\" (UniqueName: \"kubernetes.io/projected/d6cd9070-9c4b-4d84-8fde-96701967ea36-kube-api-access-87mtn\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:53.204783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.204764    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s" pod "9f08ea7a-a4f7-41bc-a450-636a96ffa72c" (UID: "9f08ea7a-a4f7-41bc-a450-636a96ffa72c"). InnerVolumeSpecName "kube-api-access-r8j4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:53.303924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.303840    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8j4s\" (UniqueName: \"kubernetes.io/projected/9f08ea7a-a4f7-41bc-a450-636a96ffa72c-kube-api-access-r8j4s\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:11:53.499023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.498990    5085 generic.go:356] "Generic (PLEG): container finished" podID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" containerID="8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87" exitCode=0
Oct 30 20:11:53.499214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.499045    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"
Oct 30 20:11:53.499214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.499074    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" event={"ID":"9f08ea7a-a4f7-41bc-a450-636a96ffa72c","Type":"ContainerDied","Data":"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"}
Oct 30 20:11:53.499214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.499130    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr" event={"ID":"9f08ea7a-a4f7-41bc-a450-636a96ffa72c","Type":"ContainerDied","Data":"64ca8edd17922a9585a0d39df078640ca86ef2fc7e1fe143d3fcce5c701a9732"}
Oct 30 20:11:53.499214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.499159    5085 scope.go:122] "RemoveContainer" containerID="8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"
Oct 30 20:11:53.500811 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.500785    5085 generic.go:356] "Generic (PLEG): container finished" podID="d6cd9070-9c4b-4d84-8fde-96701967ea36" containerID="3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad" exitCode=0
Oct 30 20:11:53.500930 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.500810    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" event={"ID":"d6cd9070-9c4b-4d84-8fde-96701967ea36","Type":"ContainerDied","Data":"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"}
Oct 30 20:11:53.500930 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.500843    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"
Oct 30 20:11:53.500930 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.500854    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h" event={"ID":"d6cd9070-9c4b-4d84-8fde-96701967ea36","Type":"ContainerDied","Data":"6ba2621e05aecb0914784d9245c6d177e9c772aa4e2c71097cd1a1f02072c937"}
Oct 30 20:11:53.516147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.516128    5085 scope.go:122] "RemoveContainer" containerID="8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"
Oct 30 20:11:53.516478 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:53.516454    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87\": container with ID starting with 8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87 not found: ID does not exist" containerID="8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"
Oct 30 20:11:53.516539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.516486    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87"} err="failed to get container status \"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87\": rpc error: code = NotFound desc = could not find container \"8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87\": container with ID starting with 8c989bc4ed2de00c0607a64fc76f653e5abfc7810b6034cf1f82b79c83b70b87 not found: ID does not exist"
Oct 30 20:11:53.516539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.516503    5085 scope.go:122] "RemoveContainer" containerID="3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"
Oct 30 20:11:53.528068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.528045    5085 scope.go:122] "RemoveContainer" containerID="3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"
Oct 30 20:11:53.528368 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:11:53.528344    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad\": container with ID starting with 3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad not found: ID does not exist" containerID="3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"
Oct 30 20:11:53.528476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.528372    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad"} err="failed to get container status \"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad\": rpc error: code = NotFound desc = could not find container \"3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad\": container with ID starting with 3c57cdb030f3e68d5e9771de8c45bb41f9f81328b9208613f96867a1aedbd9ad not found: ID does not exist"
Oct 30 20:11:53.657228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.657155    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:11:53.662270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.662245    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-84ad63-bc4fcf5dc-c9qzr"]
Oct 30 20:11:53.766678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.766642    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:11:53.769888 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:53.769863    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-84ad63-5ddbb7cfd8-b7b2h"]
Oct 30 20:11:54.632292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:54.632262    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" path="/var/lib/kubelet/pods/9f08ea7a-a4f7-41bc-a450-636a96ffa72c/volumes"
Oct 30 20:11:54.632710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:11:54.632695    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6cd9070-9c4b-4d84-8fde-96701967ea36" path="/var/lib/kubelet/pods/d6cd9070-9c4b-4d84-8fde-96701967ea36/volumes"
Oct 30 20:12:02.720825 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:02.720747    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:12:03.041687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:03.041586    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:12:03.157661 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:03.157629    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:12:08.042918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.042864    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" podUID="7548cadb-6799-4cca-81f7-aac867ca467f" containerName="payload-processing" containerID="cri-o://148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f" gracePeriod=25
Oct 30 20:12:08.158956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.158886    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" podUID="e43e8f75-e390-4387-ba59-6397bbabca5d" containerName="payload-pre-processing" containerID="cri-o://e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f" gracePeriod=25
Oct 30 20:12:08.320172 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.320140    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:12:08.377350 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.377314    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\" (UniqueName: \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\") pod \"7548cadb-6799-4cca-81f7-aac867ca467f\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") "
Oct 30 20:12:08.377542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.377481    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\") pod \"7548cadb-6799-4cca-81f7-aac867ca467f\" (UID: \"7548cadb-6799-4cca-81f7-aac867ca467f\") "
Oct 30 20:12:08.377944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.377896    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume" pod "7548cadb-6799-4cca-81f7-aac867ca467f" (UID: "7548cadb-6799-4cca-81f7-aac867ca467f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:08.379872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.379833    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p" pod "7548cadb-6799-4cca-81f7-aac867ca467f" (UID: "7548cadb-6799-4cca-81f7-aac867ca467f"). InnerVolumeSpecName "kube-api-access-jts2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:08.425270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.425245    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:12:08.478873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.478836    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\" (UniqueName: \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\") pod \"e43e8f75-e390-4387-ba59-6397bbabca5d\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") "
Oct 30 20:12:08.479094 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.478990    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\") pod \"e43e8f75-e390-4387-ba59-6397bbabca5d\" (UID: \"e43e8f75-e390-4387-ba59-6397bbabca5d\") "
Oct 30 20:12:08.479441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.479423    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts2p\" (UniqueName: \"kubernetes.io/projected/7548cadb-6799-4cca-81f7-aac867ca467f-kube-api-access-jts2p\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:08.479540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.479443    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7548cadb-6799-4cca-81f7-aac867ca467f-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:08.479540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.479474    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume" pod "e43e8f75-e390-4387-ba59-6397bbabca5d" (UID: "e43e8f75-e390-4387-ba59-6397bbabca5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:08.481435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.481406    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4" pod "e43e8f75-e390-4387-ba59-6397bbabca5d" (UID: "e43e8f75-e390-4387-ba59-6397bbabca5d"). InnerVolumeSpecName "kube-api-access-sg6b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:08.580903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.580818    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg6b4\" (UniqueName: \"kubernetes.io/projected/e43e8f75-e390-4387-ba59-6397bbabca5d-kube-api-access-sg6b4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:08.580903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.580844    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e43e8f75-e390-4387-ba59-6397bbabca5d-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:08.606624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.606591    5085 generic.go:356] "Generic (PLEG): container finished" podID="e43e8f75-e390-4387-ba59-6397bbabca5d" containerID="e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f" exitCode=0
Oct 30 20:12:08.606760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.606639    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"
Oct 30 20:12:08.606760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.606670    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" event={"ID":"e43e8f75-e390-4387-ba59-6397bbabca5d","Type":"ContainerDied","Data":"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"}
Oct 30 20:12:08.606760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.606721    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn" event={"ID":"e43e8f75-e390-4387-ba59-6397bbabca5d","Type":"ContainerDied","Data":"9769ad479ebb872008486cdfdd9b34f18bf2a0cec9759083dd5616d8a39e1ac7"}
Oct 30 20:12:08.606760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.606736    5085 scope.go:122] "RemoveContainer" containerID="e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"
Oct 30 20:12:08.608199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.608175    5085 generic.go:356] "Generic (PLEG): container finished" podID="7548cadb-6799-4cca-81f7-aac867ca467f" containerID="148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f" exitCode=0
Oct 30 20:12:08.608296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.608226    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"
Oct 30 20:12:08.608296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.608264    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" event={"ID":"7548cadb-6799-4cca-81f7-aac867ca467f","Type":"ContainerDied","Data":"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"}
Oct 30 20:12:08.608387 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.608319    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789" event={"ID":"7548cadb-6799-4cca-81f7-aac867ca467f","Type":"ContainerDied","Data":"93cde4f3d77fe8ff88bef42ac208a798342252f21169abfab4f1ccfa70f1e3c4"}
Oct 30 20:12:08.619306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.619284    5085 scope.go:122] "RemoveContainer" containerID="e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"
Oct 30 20:12:08.619798 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:08.619773    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f\": container with ID starting with e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f not found: ID does not exist" containerID="e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"
Oct 30 20:12:08.619900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.619810    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f"} err="failed to get container status \"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f\": rpc error: code = NotFound desc = could not find container \"e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f\": container with ID starting with e1deef6184865a79264e1322708e669019550a32fb85773c1978f0abc26c410f not found: ID does not exist"
Oct 30 20:12:08.619900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.619832    5085 scope.go:122] "RemoveContainer" containerID="148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"
Oct 30 20:12:08.631116 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.631094    5085 scope.go:122] "RemoveContainer" containerID="148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"
Oct 30 20:12:08.631381 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:08.631360    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f\": container with ID starting with 148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f not found: ID does not exist" containerID="148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"
Oct 30 20:12:08.631430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.631389    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f"} err="failed to get container status \"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f\": rpc error: code = NotFound desc = could not find container \"148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f\": container with ID starting with 148f96ad70ba7f572d1030eccb5b44fbc8ff0ffa3e59225a0bb94a8c43f9500f not found: ID does not exist"
Oct 30 20:12:08.764897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.764857    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:12:08.769521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.769494    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-115a78-76c6f6c696-n2789"]
Oct 30 20:12:08.872489 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.872407    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:12:08.879188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:08.879160    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-115a78-5899dfdc79-4mzzn"]
Oct 30 20:12:10.631532 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:10.631497    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7548cadb-6799-4cca-81f7-aac867ca467f" path="/var/lib/kubelet/pods/7548cadb-6799-4cca-81f7-aac867ca467f/volumes"
Oct 30 20:12:10.631953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:10.631938    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43e8f75-e390-4387-ba59-6397bbabca5d" path="/var/lib/kubelet/pods/e43e8f75-e390-4387-ba59-6397bbabca5d/volumes"
Oct 30 20:12:21.938966 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:21.938925    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:12:25.282132 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.282097    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:12:25.282601 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.282293    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" podUID="f22402bd-50b4-46fb-b209-10992ba4decd" containerName="maas-api" containerID="cri-o://0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04" gracePeriod=30
Oct 30 20:12:25.551695 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.551637    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:12:25.579618 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.579588    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\" (UniqueName: \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\") pod \"f22402bd-50b4-46fb-b209-10992ba4decd\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") "
Oct 30 20:12:25.579759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.579694    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\") pod \"f22402bd-50b4-46fb-b209-10992ba4decd\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") "
Oct 30 20:12:25.579804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.579758    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\") pod \"f22402bd-50b4-46fb-b209-10992ba4decd\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") "
Oct 30 20:12:25.579859 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.579843    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") pod \"f22402bd-50b4-46fb-b209-10992ba4decd\" (UID: \"f22402bd-50b4-46fb-b209-10992ba4decd\") "
Oct 30 20:12:25.580127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.580096    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca" pod "f22402bd-50b4-46fb-b209-10992ba4decd" (UID: "f22402bd-50b4-46fb-b209-10992ba4decd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:25.580320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.580302    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22402bd-50b4-46fb-b209-10992ba4decd-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.581829 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.581809    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg" pod "f22402bd-50b4-46fb-b209-10992ba4decd" (UID: "f22402bd-50b4-46fb-b209-10992ba4decd"). InnerVolumeSpecName "kube-api-access-jbswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:25.581951 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.581905    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls" pod "f22402bd-50b4-46fb-b209-10992ba4decd" (UID: "f22402bd-50b4-46fb-b209-10992ba4decd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:25.582020 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.581961    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs" pod "f22402bd-50b4-46fb-b209-10992ba4decd" (UID: "f22402bd-50b4-46fb-b209-10992ba4decd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:25.681922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.681886    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.681922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.681924    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbswg\" (UniqueName: \"kubernetes.io/projected/f22402bd-50b4-46fb-b209-10992ba4decd-kube-api-access-jbswg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.682227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.681940    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22402bd-50b4-46fb-b209-10992ba4decd-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:25.724654 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.724611    5085 generic.go:356] "Generic (PLEG): container finished" podID="f22402bd-50b4-46fb-b209-10992ba4decd" containerID="0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04" exitCode=0
Oct 30 20:12:25.724823 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.724652    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" event={"ID":"f22402bd-50b4-46fb-b209-10992ba4decd","Type":"ContainerDied","Data":"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"}
Oct 30 20:12:25.724823 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.724722    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp" event={"ID":"f22402bd-50b4-46fb-b209-10992ba4decd","Type":"ContainerDied","Data":"0757a3925a965eab1ac47c7057472f6cf0b79e96e61da5ecc69a8ff178e4e839"}
Oct 30 20:12:25.724823 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.724665    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"
Oct 30 20:12:25.724823 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.724751    5085 scope.go:122] "RemoveContainer" containerID="0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"
Oct 30 20:12:25.735895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.735877    5085 scope.go:122] "RemoveContainer" containerID="0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"
Oct 30 20:12:25.736156 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:25.736137    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04\": container with ID starting with 0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04 not found: ID does not exist" containerID="0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"
Oct 30 20:12:25.736203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.736165    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04"} err="failed to get container status \"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04\": rpc error: code = NotFound desc = could not find container \"0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04\": container with ID starting with 0b4ed2e687f45f2391c723e6af08eed3a230d165c18042e18be662d264683b04 not found: ID does not exist"
Oct 30 20:12:25.883052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.882967    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:12:25.889209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:25.889179    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-84ad63-5db4bdf969-k29hp"]
Oct 30 20:12:26.632507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:26.632470    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22402bd-50b4-46fb-b209-10992ba4decd" path="/var/lib/kubelet/pods/f22402bd-50b4-46fb-b209-10992ba4decd/volumes"
Oct 30 20:12:32.251019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.250983    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:12:32.251648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251631    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7548cadb-6799-4cca-81f7-aac867ca467f" containerName="payload-processing"
Oct 30 20:12:32.251648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251649    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7548cadb-6799-4cca-81f7-aac867ca467f" containerName="payload-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251666    5085 cpu_manager.go:414] "containerMap: removing container" podUID="995bc9bc-93f8-44eb-840a-70f3a133ce88" containerName="maas-api"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251671    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="995bc9bc-93f8-44eb-840a-70f3a133ce88" containerName="maas-api"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251692    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e43e8f75-e390-4387-ba59-6397bbabca5d" containerName="payload-pre-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251698    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43e8f75-e390-4387-ba59-6397bbabca5d" containerName="payload-pre-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251708    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251713    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251721    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" containerName="payload-pre-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251726    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" containerName="payload-pre-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251738    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d6cd9070-9c4b-4d84-8fde-96701967ea36" containerName="payload-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251743    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6cd9070-9c4b-4d84-8fde-96701967ea36" containerName="payload-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251750    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a091513f-11f1-4cfe-8d1a-7fc257726f91" containerName="payload-processing"
Oct 30 20:12:32.251759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251755    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a091513f-11f1-4cfe-8d1a-7fc257726f91" containerName="payload-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251775    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f22402bd-50b4-46fb-b209-10992ba4decd" containerName="maas-api"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251780    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22402bd-50b4-46fb-b209-10992ba4decd" containerName="maas-api"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251786    5085 cpu_manager.go:414] "containerMap: removing container" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251790    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251804    5085 cpu_manager.go:414] "containerMap: removing container" podUID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" containerName="payload-pre-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251808    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" containerName="payload-pre-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251818    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251822    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251829    5085 cpu_manager.go:414] "containerMap: removing container" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251833    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251924    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7548cadb-6799-4cca-81f7-aac867ca467f" containerName="payload-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251936    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="995bc9bc-93f8-44eb-840a-70f3a133ce88" containerName="maas-api"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251942    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43e8f75-e390-4387-ba59-6397bbabca5d" containerName="payload-pre-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251948    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251954    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfdcc42-daef-46b2-af7b-66527e8bcaa1" containerName="payload-pre-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251961    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6cd9070-9c4b-4d84-8fde-96701967ea36" containerName="payload-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251966    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a091513f-11f1-4cfe-8d1a-7fc257726f91" containerName="payload-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251973    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22402bd-50b4-46fb-b209-10992ba4decd" containerName="maas-api"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251979    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251986    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f08ea7a-a4f7-41bc-a450-636a96ffa72c" containerName="payload-pre-processing"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251992    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea11f0c-7bc5-4c90-8882-df523acac165" containerName="istio-proxy"
Oct 30 20:12:32.252576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.251998    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="45bb3c5d-f95c-47bc-990d-5a91e35d3d2f" containerName="istio-proxy"
Oct 30 20:12:32.256694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.256677    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.353964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.353933    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.353964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.353969    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.354223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.354146    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.354320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.354299    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v999\" (UniqueName: \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.354393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.354308    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-04679c-serving-cert\""
Oct 30 20:12:32.357757 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.357734    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:12:32.455251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.455220    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.455430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.455328    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v999\" (UniqueName: \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.455430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.455364    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.455430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.455382    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.455430 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:32.455362    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-serving-cert: secret "maas-api-e2e-praxis-b-04679c-serving-cert" not found
Oct 30 20:12:32.455665 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:32.455477    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls podName:46ffb117-2577-4e3e-a6a3-c386582f791e nodeName:}" failed. No retries permitted until 2026-10-30 20:12:32.955455745 +0000 UTC m=+2862.459859471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls") pod "maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" (UID: "46ffb117-2577-4e3e-a6a3-c386582f791e") : secret "maas-api-e2e-praxis-b-04679c-serving-cert" not found
Oct 30 20:12:32.456081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.456062    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.457766 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.457746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.515281 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.515192    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v999\" (UniqueName: \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.961445 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.961413    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:32.963801 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:32.963773    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") pod \"maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:33.170166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:33.170130    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:33.546240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:33.546215    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:12:33.547224 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:12:33.547196    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ffb117_2577_4e3e_a6a3_c386582f791e.slice/crio-ec7e4d7c5bcf9eec9e0778cdef0c9564073402df17193d2dcaff6eec9e09f774 WatchSource:0}: Error finding container ec7e4d7c5bcf9eec9e0778cdef0c9564073402df17193d2dcaff6eec9e09f774: Status 404 returned error can't find the container with id ec7e4d7c5bcf9eec9e0778cdef0c9564073402df17193d2dcaff6eec9e09f774
Oct 30 20:12:33.782485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:33.782451    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" event={"ID":"46ffb117-2577-4e3e-a6a3-c386582f791e","Type":"ContainerStarted","Data":"ec7e4d7c5bcf9eec9e0778cdef0c9564073402df17193d2dcaff6eec9e09f774"}
Oct 30 20:12:34.631871 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:34.631838    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:12:34.632332 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:34.632018    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" containerID="cri-o://aea2b390d1d2af768ed441f51e32256e170cc2e3020e33bb988f282c48f71725" gracePeriod=30
Oct 30 20:12:35.798512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:35.798471    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" event={"ID":"46ffb117-2577-4e3e-a6a3-c386582f791e","Type":"ContainerStarted","Data":"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"}
Oct 30 20:12:35.798938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:35.798553    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:35.911729 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:35.911677    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" podStartSLOduration=2.638696768 podStartE2EDuration="3.911663741s" totalImagesPullingTime="1.272966973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:35.910319026 +0000 UTC m=+2865.414722772" watchObservedRunningTime="2026-10-30 20:12:35.911663741 +0000 UTC m=+2865.416067512"
Oct 30 20:12:37.053027 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:37.052991    5085 patch_prober.go:29] interesting pod/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:12:37.053402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:37.053046    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:12:39.828969 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:39.828931    5085 generic.go:356] "Generic (PLEG): container finished" podID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerID="aea2b390d1d2af768ed441f51e32256e170cc2e3020e33bb988f282c48f71725" exitCode=0
Oct 30 20:12:39.829409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:39.829005    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerDied","Data":"aea2b390d1d2af768ed441f51e32256e170cc2e3020e33bb988f282c48f71725"}
Oct 30 20:12:39.829409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:39.829082    5085 scope.go:122] "RemoveContainer" containerID="ad898faba2b2dcdff05fbc9e066e846f3b1cc552bc3a53fd7edfcb672ee3417e"
Oct 30 20:12:39.900498 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:39.900438    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:12:40.048924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.048892    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049005    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049059    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049097    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049206    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:40.049311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049190    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049310    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:40.049418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049318    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049431    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049467    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049500    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\") pod \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\" (UID: \"5fa2dc8f-8452-4073-a50b-2f4e00abcce8\") "
Oct 30 20:12:40.049801 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049772    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:40.049893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.049822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:40.050119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.050083    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:40.050233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.050171    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.050233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.050195    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.050233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.050211    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.050233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.050226    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.051530 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.051505    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:40.051659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.051527    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:12:40.051761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.051742    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:12:40.051836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.051820    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv" pod "5fa2dc8f-8452-4073-a50b-2f4e00abcce8" (UID: "5fa2dc8f-8452-4073-a50b-2f4e00abcce8"). InnerVolumeSpecName "kube-api-access-6w7bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:40.151648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.151551    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.151648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.151595    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.151648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.151604    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.151648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.151615    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w7bv\" (UniqueName: \"kubernetes.io/projected/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-kube-api-access-6w7bv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.151648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.151625    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fa2dc8f-8452-4073-a50b-2f4e00abcce8-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:40.836546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.836517    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"
Oct 30 20:12:40.836963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.836512    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h" event={"ID":"5fa2dc8f-8452-4073-a50b-2f4e00abcce8","Type":"ContainerDied","Data":"332a8d0d39ad01c6d051ae5d9ffe06d24533bec1737c4ee43d124fd0db0c2d1e"}
Oct 30 20:12:40.836963 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:40.836662    5085 scope.go:122] "RemoveContainer" containerID="aea2b390d1d2af768ed441f51e32256e170cc2e3020e33bb988f282c48f71725"
Oct 30 20:12:41.041697 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:41.041665    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:12:41.088710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:41.088682    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-84ad63-openshift-default-69f4b8cf49-2q95h"]
Oct 30 20:12:41.811622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:41.811588    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:12:42.631364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:42.631330    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" path="/var/lib/kubelet/pods/5fa2dc8f-8452-4073-a50b-2f4e00abcce8/volumes"
Oct 30 20:12:45.312553 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:45.312518    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:12:46.462732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:46.462697    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:12:47.106207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:47.106170    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:12:47.225828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:47.225795    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:12:51.938540 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:51.938500    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:12:52.111976 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.111925    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" podUID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" containerName="payload-processing" containerID="cri-o://d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4" gracePeriod=25
Oct 30 20:12:52.226413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.226308    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" podUID="05645f4f-d82a-401a-bff5-480210fe4062" containerName="payload-pre-processing" containerID="cri-o://fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e" gracePeriod=25
Oct 30 20:12:52.388975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.388950    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:12:52.494937 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.494911    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\") pod \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") "
Oct 30 20:12:52.495083 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.495022    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\" (UniqueName: \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\") pod \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\" (UID: \"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e\") "
Oct 30 20:12:52.495288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.495263    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume" pod "9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" (UID: "9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:52.495472 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.495446    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:52.497239 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.497216    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8" pod "9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" (UID: "9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e"). InnerVolumeSpecName "kube-api-access-8zrw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:52.498903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.498888    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:12:52.596701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.596667    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\") pod \"05645f4f-d82a-401a-bff5-480210fe4062\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") "
Oct 30 20:12:52.596858 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.596763    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\" (UniqueName: \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\") pod \"05645f4f-d82a-401a-bff5-480210fe4062\" (UID: \"05645f4f-d82a-401a-bff5-480210fe4062\") "
Oct 30 20:12:52.597031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.597010    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume" pod "05645f4f-d82a-401a-bff5-480210fe4062" (UID: "05645f4f-d82a-401a-bff5-480210fe4062"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:52.597113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.597088    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrw8\" (UniqueName: \"kubernetes.io/projected/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e-kube-api-access-8zrw8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:52.597113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.597105    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05645f4f-d82a-401a-bff5-480210fe4062-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:52.598833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.598810    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4" pod "05645f4f-d82a-401a-bff5-480210fe4062" (UID: "05645f4f-d82a-401a-bff5-480210fe4062"). InnerVolumeSpecName "kube-api-access-5chj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:52.697922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.697893    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5chj4\" (UniqueName: \"kubernetes.io/projected/05645f4f-d82a-401a-bff5-480210fe4062-kube-api-access-5chj4\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:12:52.918897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.918803    5085 generic.go:356] "Generic (PLEG): container finished" podID="05645f4f-d82a-401a-bff5-480210fe4062" containerID="fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e" exitCode=0
Oct 30 20:12:52.918897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.918856    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"
Oct 30 20:12:52.918897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.918889    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" event={"ID":"05645f4f-d82a-401a-bff5-480210fe4062","Type":"ContainerDied","Data":"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"}
Oct 30 20:12:52.919187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.918941    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf" event={"ID":"05645f4f-d82a-401a-bff5-480210fe4062","Type":"ContainerDied","Data":"8041d8a944d041929412b832659a8133def2c549af0a0b955060843ac7ef42ff"}
Oct 30 20:12:52.919187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.918967    5085 scope.go:122] "RemoveContainer" containerID="fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"
Oct 30 20:12:52.920599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.920558    5085 generic.go:356] "Generic (PLEG): container finished" podID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" containerID="d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4" exitCode=0
Oct 30 20:12:52.920734 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.920610    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" event={"ID":"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e","Type":"ContainerDied","Data":"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"}
Oct 30 20:12:52.920734 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.920665    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25" event={"ID":"9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e","Type":"ContainerDied","Data":"9c4d6c1bedc52b6d19a6eb6bcc17be3f95455bf34b67139191fcd2f1b531fd83"}
Oct 30 20:12:52.920734 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.920670    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"
Oct 30 20:12:52.929432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.929411    5085 scope.go:122] "RemoveContainer" containerID="fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"
Oct 30 20:12:52.929722 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:52.929703    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e\": container with ID starting with fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e not found: ID does not exist" containerID="fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"
Oct 30 20:12:52.929790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.929738    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e"} err="failed to get container status \"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e\": rpc error: code = NotFound desc = could not find container \"fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e\": container with ID starting with fa38aae89d8d57e3ed0ec6be43388002a8092b16f3dd9e52b2639c0150f2109e not found: ID does not exist"
Oct 30 20:12:52.929790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.929754    5085 scope.go:122] "RemoveContainer" containerID="d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"
Oct 30 20:12:52.938584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.938544    5085 scope.go:122] "RemoveContainer" containerID="d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"
Oct 30 20:12:52.938882 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:12:52.938841    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4\": container with ID starting with d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4 not found: ID does not exist" containerID="d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"
Oct 30 20:12:52.938935 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:52.938867    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4"} err="failed to get container status \"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4\": rpc error: code = NotFound desc = could not find container \"d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4\": container with ID starting with d5113ba04f707802c5a764b7f940f056696af32c2cfbf65585591838d7be03a4 not found: ID does not exist"
Oct 30 20:12:53.077232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:53.077201    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:12:53.081032 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:53.081007    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a3c171-66975dc446-jbl25"]
Oct 30 20:12:53.187424 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:53.187392    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:12:53.191750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:53.191722    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a3c171-789cccdb7-hv9gf"]
Oct 30 20:12:54.005146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:54.005112    5085 scope.go:122] "RemoveContainer" containerID="1d78f4fda9727b4fce37a3023381920c69bf8b4f0a06da6793f34ffb6e2d6e90"
Oct 30 20:12:54.631207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:54.631176    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05645f4f-d82a-401a-bff5-480210fe4062" path="/var/lib/kubelet/pods/05645f4f-d82a-401a-bff5-480210fe4062/volumes"
Oct 30 20:12:54.631660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:12:54.631644    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" path="/var/lib/kubelet/pods/9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e/volumes"
Oct 30 20:13:00.334069 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.334029    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" podUID="ac00388d-39fd-4e68-a249-3430e4591667" containerName="main" containerID="cri-o://7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc" gracePeriod=45
Oct 30 20:13:00.600030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.599970    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:13:00.688474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688445    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688534    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688608    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688634    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688674    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\" (UniqueName: \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688778    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\") pod \"ac00388d-39fd-4e68-a249-3430e4591667\" (UID: \"ac00388d-39fd-4e68-a249-3430e4591667\") "
Oct 30 20:13:00.688869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688840    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:00.689021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688892    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:00.689021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.688922    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:00.689251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.689232    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.689251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.689251    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.689408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.689260    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.690784 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.690763    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:13:00.690916 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.690901    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:00.691012 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.690993    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6" pod "ac00388d-39fd-4e68-a249-3430e4591667" (UID: "ac00388d-39fd-4e68-a249-3430e4591667"). InnerVolumeSpecName "kube-api-access-9ghz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:00.790882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.790845    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac00388d-39fd-4e68-a249-3430e4591667-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.790882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.790878    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac00388d-39fd-4e68-a249-3430e4591667-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.791085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.790889    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ghz6\" (UniqueName: \"kubernetes.io/projected/ac00388d-39fd-4e68-a249-3430e4591667-kube-api-access-9ghz6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:00.974806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.974770    5085 generic.go:356] "Generic (PLEG): container finished" podID="ac00388d-39fd-4e68-a249-3430e4591667" containerID="7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc" exitCode=0
Oct 30 20:13:00.974993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.974820    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"
Oct 30 20:13:00.974993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.974854    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" event={"ID":"ac00388d-39fd-4e68-a249-3430e4591667","Type":"ContainerDied","Data":"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"}
Oct 30 20:13:00.974993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.974911    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" event={"ID":"ac00388d-39fd-4e68-a249-3430e4591667","Type":"ContainerDied","Data":"2ee689b7f5d84ddf40c5dd441b4cf3fe713530462f3b85f7e54fdd24015f7c08"}
Oct 30 20:13:00.974993 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.974929    5085 scope.go:122] "RemoveContainer" containerID="7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"
Oct 30 20:13:00.986158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.986140    5085 scope.go:122] "RemoveContainer" containerID="7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"
Oct 30 20:13:00.986414 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:13:00.986389    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc\": container with ID starting with 7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc not found: ID does not exist" containerID="7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"
Oct 30 20:13:00.986516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:00.986418    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc"} err="failed to get container status \"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc\": rpc error: code = NotFound desc = could not find container \"7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc\": container with ID starting with 7fde5e8fadd6c4ef8b5beeed0d5194053c095df52866a3098fca513c321068bc not found: ID does not exist"
Oct 30 20:13:01.132115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:01.132086    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:13:01.135066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:01.135037    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq"]
Oct 30 20:13:01.519972 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:01.519932    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-eaa6d2/ipp-route-eaa6d2-kserve-5db5dc45bf-zktvq" podUID="ac00388d-39fd-4e68-a249-3430e4591667" containerName="main" probeResult="failure" output="Get \"https://10.132.0.199:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:13:02.631348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:02.631312    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac00388d-39fd-4e68-a249-3430e4591667" path="/var/lib/kubelet/pods/ac00388d-39fd-4e68-a249-3430e4591667/volumes"
Oct 30 20:13:07.164049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.164009    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:13:07.164514 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.164255    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" podUID="33f839db-30a0-4ec2-ac02-e477a104e511" containerName="maas-api" containerID="cri-o://746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a" gracePeriod=30
Oct 30 20:13:07.440851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.440828    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:13:07.579024 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.578986    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\") pod \"33f839db-30a0-4ec2-ac02-e477a104e511\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") "
Oct 30 20:13:07.579282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.579118    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\" (UniqueName: \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\") pod \"33f839db-30a0-4ec2-ac02-e477a104e511\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") "
Oct 30 20:13:07.579282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.579191    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\") pod \"33f839db-30a0-4ec2-ac02-e477a104e511\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") "
Oct 30 20:13:07.579282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.579265    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") pod \"33f839db-30a0-4ec2-ac02-e477a104e511\" (UID: \"33f839db-30a0-4ec2-ac02-e477a104e511\") "
Oct 30 20:13:07.579453 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.579362    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca" pod "33f839db-30a0-4ec2-ac02-e477a104e511" (UID: "33f839db-30a0-4ec2-ac02-e477a104e511"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:07.579754 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.579729    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33f839db-30a0-4ec2-ac02-e477a104e511-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.581442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.581417    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf" pod "33f839db-30a0-4ec2-ac02-e477a104e511" (UID: "33f839db-30a0-4ec2-ac02-e477a104e511"). InnerVolumeSpecName "kube-api-access-78ftf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:07.581783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.581758    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls" pod "33f839db-30a0-4ec2-ac02-e477a104e511" (UID: "33f839db-30a0-4ec2-ac02-e477a104e511"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:07.581897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.581853    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs" pod "33f839db-30a0-4ec2-ac02-e477a104e511" (UID: "33f839db-30a0-4ec2-ac02-e477a104e511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:07.680895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.680778    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.680895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.680815    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ftf\" (UniqueName: \"kubernetes.io/projected/33f839db-30a0-4ec2-ac02-e477a104e511-kube-api-access-78ftf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.680895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:07.680833    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33f839db-30a0-4ec2-ac02-e477a104e511-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:08.019347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.019260    5085 generic.go:356] "Generic (PLEG): container finished" podID="33f839db-30a0-4ec2-ac02-e477a104e511" containerID="746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a" exitCode=0
Oct 30 20:13:08.019347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.019312    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"
Oct 30 20:13:08.019535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.019340    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" event={"ID":"33f839db-30a0-4ec2-ac02-e477a104e511","Type":"ContainerDied","Data":"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"}
Oct 30 20:13:08.019535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.019393    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp" event={"ID":"33f839db-30a0-4ec2-ac02-e477a104e511","Type":"ContainerDied","Data":"39a66ff571df207d5856a7c5d4aa1d001afccb3b26ee1ea8dcb72faea1a05cee"}
Oct 30 20:13:08.019535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.019415    5085 scope.go:122] "RemoveContainer" containerID="746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"
Oct 30 20:13:08.034232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.034210    5085 scope.go:122] "RemoveContainer" containerID="746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"
Oct 30 20:13:08.034511 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:13:08.034491    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a\": container with ID starting with 746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a not found: ID does not exist" containerID="746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"
Oct 30 20:13:08.034586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.034520    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a"} err="failed to get container status \"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a\": rpc error: code = NotFound desc = could not find container \"746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a\": container with ID starting with 746cb60bd9acfa11a34a2d4e3a593b2f98e608701a157d12e8fa34360946104a not found: ID does not exist"
Oct 30 20:13:08.177687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.177659    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:13:08.224402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.224371    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-115a78-5c959cf779-d5pxp"]
Oct 30 20:13:08.631398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:08.631362    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f839db-30a0-4ec2-ac02-e477a104e511" path="/var/lib/kubelet/pods/33f839db-30a0-4ec2-ac02-e477a104e511/volumes"
Oct 30 20:13:21.939881 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:13:21.939836    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:13:22.935748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:22.935716    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:13:44.275116 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.274727    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:13:44.278453 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.275899    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" podUID="40dc596b-34f4-4871-83fc-066df0d09776" containerName="maas-api" containerID="cri-o://223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb" gracePeriod=30
Oct 30 20:13:44.547037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.546977    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:13:44.699525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.699489    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\") pod \"40dc596b-34f4-4871-83fc-066df0d09776\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") "
Oct 30 20:13:44.699751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.699654    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\") pod \"40dc596b-34f4-4871-83fc-066df0d09776\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") "
Oct 30 20:13:44.699751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.699734    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") pod \"40dc596b-34f4-4871-83fc-066df0d09776\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") "
Oct 30 20:13:44.699870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.699784    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\" (UniqueName: \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\") pod \"40dc596b-34f4-4871-83fc-066df0d09776\" (UID: \"40dc596b-34f4-4871-83fc-066df0d09776\") "
Oct 30 20:13:44.700169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.700119    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca" pod "40dc596b-34f4-4871-83fc-066df0d09776" (UID: "40dc596b-34f4-4871-83fc-066df0d09776"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:44.700285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.700273    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40dc596b-34f4-4871-83fc-066df0d09776-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:44.702134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.702097    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs" pod "40dc596b-34f4-4871-83fc-066df0d09776" (UID: "40dc596b-34f4-4871-83fc-066df0d09776"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:44.702292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.702268    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls" pod "40dc596b-34f4-4871-83fc-066df0d09776" (UID: "40dc596b-34f4-4871-83fc-066df0d09776"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:44.702606 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.702552    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q" pod "40dc596b-34f4-4871-83fc-066df0d09776" (UID: "40dc596b-34f4-4871-83fc-066df0d09776"). InnerVolumeSpecName "kube-api-access-j849q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:44.801395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.801309    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j849q\" (UniqueName: \"kubernetes.io/projected/40dc596b-34f4-4871-83fc-066df0d09776-kube-api-access-j849q\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:44.801395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.801348    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:44.801395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:44.801364    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40dc596b-34f4-4871-83fc-066df0d09776-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:13:45.269572 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.269535    5085 generic.go:356] "Generic (PLEG): container finished" podID="40dc596b-34f4-4871-83fc-066df0d09776" containerID="223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb" exitCode=0
Oct 30 20:13:45.269751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.269617    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"
Oct 30 20:13:45.269751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.269616    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" event={"ID":"40dc596b-34f4-4871-83fc-066df0d09776","Type":"ContainerDied","Data":"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"}
Oct 30 20:13:45.269751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.269672    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp" event={"ID":"40dc596b-34f4-4871-83fc-066df0d09776","Type":"ContainerDied","Data":"42b84c3a17a33588e1c79310a86ee8d8579811bf7c3dd6d72e710c0cb6c0caed"}
Oct 30 20:13:45.269751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.269695    5085 scope.go:122] "RemoveContainer" containerID="223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"
Oct 30 20:13:45.280900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.280751    5085 scope.go:122] "RemoveContainer" containerID="223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"
Oct 30 20:13:45.281207 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:13:45.281105    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb\": container with ID starting with 223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb not found: ID does not exist" containerID="223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"
Oct 30 20:13:45.281207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.281132    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb"} err="failed to get container status \"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb\": rpc error: code = NotFound desc = could not find container \"223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb\": container with ID starting with 223bcbd3d3bebbc119754cd2771e25adccf139de2a05aa24cbdc6ff6534d2dcb not found: ID does not exist"
Oct 30 20:13:45.427322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.427283    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:13:45.474271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:45.474239    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a3c171-7fc747dbfb-s9fqp"]
Oct 30 20:13:46.631978 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:46.631940    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40dc596b-34f4-4871-83fc-066df0d09776" path="/var/lib/kubelet/pods/40dc596b-34f4-4871-83fc-066df0d09776/volumes"
Oct 30 20:13:51.944344 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:13:51.944299    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:13:54.063710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:13:54.063681    5085 scope.go:122] "RemoveContainer" containerID="c62d1978beec8165ac6672d28361576e2be76c1cb3eb79a6bde8779de9eb7f35"
Oct 30 20:14:01.259234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:01.259196    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:14:01.548797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:01.548718    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:14:01.682792 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:01.682759    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:14:06.549015 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.548958    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" podUID="eea45404-def4-49c2-b7d4-783608fb65cf" containerName="payload-processing" containerID="cri-o://9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113" gracePeriod=25
Oct 30 20:14:06.683050 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.683014    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" podUID="0f9093f0-44c2-4967-8279-19b55d3c102f" containerName="payload-pre-processing" containerID="cri-o://0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f" gracePeriod=25
Oct 30 20:14:06.815488 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.815462    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:14:06.842590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.842550    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\") pod \"eea45404-def4-49c2-b7d4-783608fb65cf\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") "
Oct 30 20:14:06.842743 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.842692    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\" (UniqueName: \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\") pod \"eea45404-def4-49c2-b7d4-783608fb65cf\" (UID: \"eea45404-def4-49c2-b7d4-783608fb65cf\") "
Oct 30 20:14:06.842970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.842942    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume" pod "eea45404-def4-49c2-b7d4-783608fb65cf" (UID: "eea45404-def4-49c2-b7d4-783608fb65cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:06.843223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.843200    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eea45404-def4-49c2-b7d4-783608fb65cf-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:06.844741 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.844711    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg" pod "eea45404-def4-49c2-b7d4-783608fb65cf" (UID: "eea45404-def4-49c2-b7d4-783608fb65cf"). InnerVolumeSpecName "kube-api-access-bfpgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:06.944497 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.944469    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpgg\" (UniqueName: \"kubernetes.io/projected/eea45404-def4-49c2-b7d4-783608fb65cf-kube-api-access-bfpgg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:06.947541 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:06.947515    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:14:07.045171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.045134    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\") pod \"0f9093f0-44c2-4967-8279-19b55d3c102f\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") "
Oct 30 20:14:07.045383 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.045341    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\" (UniqueName: \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\") pod \"0f9093f0-44c2-4967-8279-19b55d3c102f\" (UID: \"0f9093f0-44c2-4967-8279-19b55d3c102f\") "
Oct 30 20:14:07.045524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.045501    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume" pod "0f9093f0-44c2-4967-8279-19b55d3c102f" (UID: "0f9093f0-44c2-4967-8279-19b55d3c102f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:07.045842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.045824    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9093f0-44c2-4967-8279-19b55d3c102f-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:07.047480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.047461    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t" pod "0f9093f0-44c2-4967-8279-19b55d3c102f" (UID: "0f9093f0-44c2-4967-8279-19b55d3c102f"). InnerVolumeSpecName "kube-api-access-kt55t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:07.146836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.146761    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt55t\" (UniqueName: \"kubernetes.io/projected/0f9093f0-44c2-4967-8279-19b55d3c102f-kube-api-access-kt55t\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:07.428495 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.428461    5085 generic.go:356] "Generic (PLEG): container finished" podID="0f9093f0-44c2-4967-8279-19b55d3c102f" containerID="0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f" exitCode=0
Oct 30 20:14:07.428722 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.428599    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"
Oct 30 20:14:07.428722 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.428606    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" event={"ID":"0f9093f0-44c2-4967-8279-19b55d3c102f","Type":"ContainerDied","Data":"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"}
Oct 30 20:14:07.428722 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.428671    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn" event={"ID":"0f9093f0-44c2-4967-8279-19b55d3c102f","Type":"ContainerDied","Data":"02299e20596c0eb7f03fc562f64926ee22c29c9c7a92714e86b041cb5c28c0d8"}
Oct 30 20:14:07.428722 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.428702    5085 scope.go:122] "RemoveContainer" containerID="0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"
Oct 30 20:14:07.430268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.430245    5085 generic.go:356] "Generic (PLEG): container finished" podID="eea45404-def4-49c2-b7d4-783608fb65cf" containerID="9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113" exitCode=0
Oct 30 20:14:07.430400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.430290    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" event={"ID":"eea45404-def4-49c2-b7d4-783608fb65cf","Type":"ContainerDied","Data":"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"}
Oct 30 20:14:07.430400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.430312    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"
Oct 30 20:14:07.430400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.430330    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr" event={"ID":"eea45404-def4-49c2-b7d4-783608fb65cf","Type":"ContainerDied","Data":"2740a3f707c81949b45d522369d01fcc330bf79e2270a11d3a52d6f46c87d19c"}
Oct 30 20:14:07.439111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.439094    5085 scope.go:122] "RemoveContainer" containerID="0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"
Oct 30 20:14:07.439401 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:07.439383    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f\": container with ID starting with 0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f not found: ID does not exist" containerID="0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"
Oct 30 20:14:07.439455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.439409    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f"} err="failed to get container status \"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f\": rpc error: code = NotFound desc = could not find container \"0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f\": container with ID starting with 0a8e2893d05f334beb8696c3f0ff0715928da95461a3fdde581b937295abf05f not found: ID does not exist"
Oct 30 20:14:07.439455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.439428    5085 scope.go:122] "RemoveContainer" containerID="9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"
Oct 30 20:14:07.449985 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.449909    5085 scope.go:122] "RemoveContainer" containerID="9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"
Oct 30 20:14:07.452111 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:07.452080    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113\": container with ID starting with 9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113 not found: ID does not exist" containerID="9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"
Oct 30 20:14:07.452207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.452119    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113"} err="failed to get container status \"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113\": rpc error: code = NotFound desc = could not find container \"9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113\": container with ID starting with 9374a18f731efda208ae1d7a595e1ac10d857467c84e624a915c4d49e727f113 not found: ID does not exist"
Oct 30 20:14:07.585324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.585294    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:14:07.589523 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.589500    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-e6b14d-768b64465f-8w6xn"]
Oct 30 20:14:07.699747 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.699667    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:14:07.707933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:07.707904    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-e6b14d-d77c786b8-7pwvr"]
Oct 30 20:14:08.630713 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:08.630685    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9093f0-44c2-4967-8279-19b55d3c102f" path="/var/lib/kubelet/pods/0f9093f0-44c2-4967-8279-19b55d3c102f/volumes"
Oct 30 20:14:08.631135 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:08.631111    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea45404-def4-49c2-b7d4-783608fb65cf" path="/var/lib/kubelet/pods/eea45404-def4-49c2-b7d4-783608fb65cf/volumes"
Oct 30 20:14:16.928638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:16.928597    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:14:17.046239 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:17.046206    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:14:21.003987 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.003949    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:14:21.004485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.004176    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" podUID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" containerName="maas-api" containerID="cri-o://8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9" gracePeriod=30
Oct 30 20:14:21.280166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.280134    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:14:21.302241 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302211    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:14:21.302812 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302798    5085 cpu_manager.go:414] "containerMap: removing container" podUID="40dc596b-34f4-4871-83fc-066df0d09776" containerName="maas-api"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302813    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40dc596b-34f4-4871-83fc-066df0d09776" containerName="maas-api"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302833    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302838    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302851    5085 cpu_manager.go:414] "containerMap: removing container" podUID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" containerName="payload-processing"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302855    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" containerName="payload-processing"
Oct 30 20:14:21.302870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302871    5085 cpu_manager.go:414] "containerMap: removing container" podUID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" containerName="maas-api"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302877    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" containerName="maas-api"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302887    5085 cpu_manager.go:414] "containerMap: removing container" podUID="33f839db-30a0-4ec2-ac02-e477a104e511" containerName="maas-api"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302892    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f839db-30a0-4ec2-ac02-e477a104e511" containerName="maas-api"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302900    5085 cpu_manager.go:414] "containerMap: removing container" podUID="eea45404-def4-49c2-b7d4-783608fb65cf" containerName="payload-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302905    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea45404-def4-49c2-b7d4-783608fb65cf" containerName="payload-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302925    5085 cpu_manager.go:414] "containerMap: removing container" podUID="05645f4f-d82a-401a-bff5-480210fe4062" containerName="payload-pre-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302930    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05645f4f-d82a-401a-bff5-480210fe4062" containerName="payload-pre-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302939    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ac00388d-39fd-4e68-a249-3430e4591667" containerName="main"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302943    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac00388d-39fd-4e68-a249-3430e4591667" containerName="main"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302962    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0f9093f0-44c2-4967-8279-19b55d3c102f" containerName="payload-pre-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.302966    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9093f0-44c2-4967-8279-19b55d3c102f" containerName="payload-pre-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303053    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303064    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5cf2cb-ef10-4542-8b21-d4efcdf8b71e" containerName="payload-processing"
Oct 30 20:14:21.303067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303070    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" containerName="maas-api"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303076    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f839db-30a0-4ec2-ac02-e477a104e511" containerName="maas-api"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303083    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea45404-def4-49c2-b7d4-783608fb65cf" containerName="payload-processing"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303090    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="05645f4f-d82a-401a-bff5-480210fe4062" containerName="payload-pre-processing"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303097    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac00388d-39fd-4e68-a249-3430e4591667" containerName="main"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303104    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9093f0-44c2-4967-8279-19b55d3c102f" containerName="payload-pre-processing"
Oct 30 20:14:21.303630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.303110    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="40dc596b-34f4-4871-83fc-066df0d09776" containerName="maas-api"
Oct 30 20:14:21.306494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.306476    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.357365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.357314    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d5d3ba\""
Oct 30 20:14:21.357365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.357343    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d5d3ba-dockercfg-k8kfw\""
Oct 30 20:14:21.361430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.361407    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:14:21.407533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.407499    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\" (UniqueName: \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\") pod \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") "
Oct 30 20:14:21.407703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.407617    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") pod \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") "
Oct 30 20:14:21.407703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.407678    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\") pod \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") "
Oct 30 20:14:21.407815 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.407760    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\") pod \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\" (UID: \"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6\") "
Oct 30 20:14:21.408260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.408043    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5rpk\" (UniqueName: \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.408260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.408151    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca" pod "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" (UID: "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:21.408456 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.408395    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.408573 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.408531    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.409929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.409905    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls" pod "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" (UID: "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:21.410100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.410024    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d" pod "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" (UID: "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6"). InnerVolumeSpecName "kube-api-access-9cv4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:21.410288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.410270    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs" pod "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" (UID: "bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:21.456104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.456062    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:14:21.457091 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.457055    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.457091 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.457085    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.457329 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.457312    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa2dc8f-8452-4073-a50b-2f4e00abcce8" containerName="istio-proxy"
Oct 30 20:14:21.462846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.462823    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.509832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.509757    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.509832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.509822    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5rpk\" (UniqueName: \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.510054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.510000    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.510054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.510021    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cv4d\" (UniqueName: \"kubernetes.io/projected/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-kube-api-access-9cv4d\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.510054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.510039    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:21.510210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.510080    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:14:21.510736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.510710    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.533404 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.533368    5085 generic.go:356] "Generic (PLEG): container finished" podID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" containerID="8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9" exitCode=0
Oct 30 20:14:21.533600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.533444    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"
Oct 30 20:14:21.533600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.533493    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" event={"ID":"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6","Type":"ContainerDied","Data":"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"}
Oct 30 20:14:21.533600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.533539    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq" event={"ID":"bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6","Type":"ContainerDied","Data":"c2b26556e06aa701505a7d976d68296da52f1af64efb429ea18523331b511dcf"}
Oct 30 20:14:21.533772 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.533606    5085 scope.go:122] "RemoveContainer" containerID="8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"
Oct 30 20:14:21.547003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.546982    5085 scope.go:122] "RemoveContainer" containerID="8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"
Oct 30 20:14:21.547332 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:21.547306    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9\": container with ID starting with 8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9 not found: ID does not exist" containerID="8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"
Oct 30 20:14:21.547423 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.547337    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9"} err="failed to get container status \"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9\": rpc error: code = NotFound desc = could not find container \"8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9\": container with ID starting with 8aa9a7a9dae07046b6ab73ea584c859c06ebd7e83baf3b132316b613f39deda9 not found: ID does not exist"
Oct 30 20:14:21.570251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.570217    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5rpk\" (UniqueName: \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\") pod \"payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.610768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.610741    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.610949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.610929    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jpqm\" (UniqueName: \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.621696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.621671    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:21.712885 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.712845    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jpqm\" (UniqueName: \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.713063 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.712965    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.714408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.714377    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.725790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.725766    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:14:21.730588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.730547    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-e6b14d-5c78c969f7-slplq"]
Oct 30 20:14:21.771789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.771716    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jpqm\" (UniqueName: \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\") pod \"payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.785558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.785537    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:21.929849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.929795    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing" containerID="cri-o://aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0" gracePeriod=25
Oct 30 20:14:21.940026 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:21.939996    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:14:21.994802 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:21.994768    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:14:22.004283 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:14:22.003695    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714cde8b_c850_44ab_9fce_c2675c3e28b4.slice/crio-a2b1246f54d0c54a1d4330bfb561a1de0b7ed05194b3c6bd0e66a8c0e185a9cd WatchSource:0}: Error finding container a2b1246f54d0c54a1d4330bfb561a1de0b7ed05194b3c6bd0e66a8c0e185a9cd: Status 404 returned error can't find the container with id a2b1246f54d0c54a1d4330bfb561a1de0b7ed05194b3c6bd0e66a8c0e185a9cd
Oct 30 20:14:22.046599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.046539    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing" containerID="cri-o://c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb" gracePeriod=25
Oct 30 20:14:22.157164 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.157135    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:14:22.159187 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:14:22.159161    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3df9cda_6022_41ea_9a7b_9b267dcb7828.slice/crio-a8b68414951c0659071ac3c774b6d337717cbdd7dcc7d9b7188652a5587e4b95 WatchSource:0}: Error finding container a8b68414951c0659071ac3c774b6d337717cbdd7dcc7d9b7188652a5587e4b95: Status 404 returned error can't find the container with id a8b68414951c0659071ac3c774b6d337717cbdd7dcc7d9b7188652a5587e4b95
Oct 30 20:14:22.546628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.546593    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" event={"ID":"a3df9cda-6022-41ea-9a7b-9b267dcb7828","Type":"ContainerStarted","Data":"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"}
Oct 30 20:14:22.546836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.546652    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" event={"ID":"a3df9cda-6022-41ea-9a7b-9b267dcb7828","Type":"ContainerStarted","Data":"a8b68414951c0659071ac3c774b6d337717cbdd7dcc7d9b7188652a5587e4b95"}
Oct 30 20:14:22.546836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.546684    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:22.548092 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.548069    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" event={"ID":"714cde8b-c850-44ab-9fce-c2675c3e28b4","Type":"ContainerStarted","Data":"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"}
Oct 30 20:14:22.548213 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.548106    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" event={"ID":"714cde8b-c850-44ab-9fce-c2675c3e28b4","Type":"ContainerStarted","Data":"a2b1246f54d0c54a1d4330bfb561a1de0b7ed05194b3c6bd0e66a8c0e185a9cd"}
Oct 30 20:14:22.548213 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.548143    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:22.631598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.631551    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6" path="/var/lib/kubelet/pods/bb5b5b5d-39a1-4de6-af96-2c3fd5fb5ad6/volumes"
Oct 30 20:14:22.658753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.658707    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" podStartSLOduration=1.658692134 podStartE2EDuration="1.658692134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:22.657618456 +0000 UTC m=+2972.162022196" watchObservedRunningTime="2026-10-30 20:14:22.658692134 +0000 UTC m=+2972.163095880"
Oct 30 20:14:22.772555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:22.772503    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" podStartSLOduration=1.772489134 podStartE2EDuration="1.772489134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:22.768784503 +0000 UTC m=+2972.273188250" watchObservedRunningTime="2026-10-30 20:14:22.772489134 +0000 UTC m=+2972.276892881"
Oct 30 20:14:26.078658 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:26.078618    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:26.078658 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:26.078649    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:31.076051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.076009    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"]
Oct 30 20:14:31.083773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.083744    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.115494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.115466    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrzxq\" (UniqueName: \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.115625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.115590    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.115691 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.115631    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.128545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.128520    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"]
Oct 30 20:14:31.171629 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.171601    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:31.171778 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.171672    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:14:31.217037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.217005    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.217217 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.217049    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.217217 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.217132    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrzxq\" (UniqueName: \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.217850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.217819    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.219320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.219299    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.275282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.275249    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrzxq\" (UniqueName: \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\") pod \"maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.397816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.397731    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:31.768105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:31.768076    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"]
Oct 30 20:14:31.769190 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:14:31.769163    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5408b87c_83a3_4b2f_bb6e_0eb34f792b36.slice/crio-2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f WatchSource:0}: Error finding container 2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f: Status 404 returned error can't find the container with id 2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f
Oct 30 20:14:32.616037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:32.616001    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf" event={"ID":"5408b87c-83a3-4b2f-bb6e-0eb34f792b36","Type":"ContainerStarted","Data":"dc042f5a2d5d33164f4935fb591fe9a6cb489c579e40feb8b98d448c9814f561"}
Oct 30 20:14:32.616458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:32.616069    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf" event={"ID":"5408b87c-83a3-4b2f-bb6e-0eb34f792b36","Type":"ContainerStarted","Data":"2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f"}
Oct 30 20:14:32.727445 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:32.727383    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf" podStartSLOduration=1.7273630519999998 podStartE2EDuration="1.727363052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:32.726546986 +0000 UTC m=+2982.230950736" watchObservedRunningTime="2026-10-30 20:14:32.727363052 +0000 UTC m=+2982.231766802"
Oct 30 20:14:33.558257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:33.558227    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:14:33.558417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:33.558296    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:14:33.624068 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:33.624036    5085 generic.go:356] "Generic (PLEG): container finished" podID="5408b87c-83a3-4b2f-bb6e-0eb34f792b36" containerID="dc042f5a2d5d33164f4935fb591fe9a6cb489c579e40feb8b98d448c9814f561" exitCode=0
Oct 30 20:14:33.624455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:33.624110    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf" event={"ID":"5408b87c-83a3-4b2f-bb6e-0eb34f792b36","Type":"ContainerDied","Data":"dc042f5a2d5d33164f4935fb591fe9a6cb489c579e40feb8b98d448c9814f561"}
Oct 30 20:14:34.766214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.766189    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:34.855175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.855143    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\") pod \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") "
Oct 30 20:14:34.855382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.855343    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\" (UniqueName: \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\") pod \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") "
Oct 30 20:14:34.855382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.855381    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\") pod \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\" (UID: \"5408b87c-83a3-4b2f-bb6e-0eb34f792b36\") "
Oct 30 20:14:34.855817 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.855785    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca" pod "5408b87c-83a3-4b2f-bb6e-0eb34f792b36" (UID: "5408b87c-83a3-4b2f-bb6e-0eb34f792b36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:34.857244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.857219    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca" pod "5408b87c-83a3-4b2f-bb6e-0eb34f792b36" (UID: "5408b87c-83a3-4b2f-bb6e-0eb34f792b36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:34.857413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.857397    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq" pod "5408b87c-83a3-4b2f-bb6e-0eb34f792b36" (UID: "5408b87c-83a3-4b2f-bb6e-0eb34f792b36"). InnerVolumeSpecName "kube-api-access-lrzxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:34.957125 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.957095    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrzxq\" (UniqueName: \"kubernetes.io/projected/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-kube-api-access-lrzxq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:34.957125 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.957123    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:34.957325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:34.957134    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5408b87c-83a3-4b2f-bb6e-0eb34f792b36-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:35.640056 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:35.640025    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"
Oct 30 20:14:35.640251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:35.640066    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf" event={"ID":"5408b87c-83a3-4b2f-bb6e-0eb34f792b36","Type":"ContainerDied","Data":"2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f"}
Oct 30 20:14:35.640251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:35.640116    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d43d6327ee3d9a91430724661d9d3d2abcf04b40f205f6720c2f955b4c5484f"
Oct 30 20:14:36.078901 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:36.078845    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:36.079334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:36.079004    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:14:42.202872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.202796    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:14:42.297533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.297509    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:14:42.336840 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.336809    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\") pod \"c7918962-8970-4916-960b-a31341090029\" (UID: \"c7918962-8970-4916-960b-a31341090029\") "
Oct 30 20:14:42.337007 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.336892    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\" (UniqueName: \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\") pod \"c7918962-8970-4916-960b-a31341090029\" (UID: \"c7918962-8970-4916-960b-a31341090029\") "
Oct 30 20:14:42.337261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.337236    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume" pod "c7918962-8970-4916-960b-a31341090029" (UID: "c7918962-8970-4916-960b-a31341090029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:42.337371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.337354    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7918962-8970-4916-960b-a31341090029-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:42.339201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.339173    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn" pod "c7918962-8970-4916-960b-a31341090029" (UID: "c7918962-8970-4916-960b-a31341090029"). InnerVolumeSpecName "kube-api-access-56csn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:42.438350 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.438312    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\" (UniqueName: \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\") pod \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") "
Oct 30 20:14:42.438552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.438516    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\") pod \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\" (UID: \"ec296583-5915-4c7a-8ea7-96d3a7f6647d\") "
Oct 30 20:14:42.438914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.438888    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume" pod "ec296583-5915-4c7a-8ea7-96d3a7f6647d" (UID: "ec296583-5915-4c7a-8ea7-96d3a7f6647d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:42.439047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.439026    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56csn\" (UniqueName: \"kubernetes.io/projected/c7918962-8970-4916-960b-a31341090029-kube-api-access-56csn\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:42.439126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.439050    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec296583-5915-4c7a-8ea7-96d3a7f6647d-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:42.440904 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.440882    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl" pod "ec296583-5915-4c7a-8ea7-96d3a7f6647d" (UID: "ec296583-5915-4c7a-8ea7-96d3a7f6647d"). InnerVolumeSpecName "kube-api-access-d9zjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:42.539921 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.539838    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9zjl\" (UniqueName: \"kubernetes.io/projected/ec296583-5915-4c7a-8ea7-96d3a7f6647d-kube-api-access-d9zjl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:14:42.685045 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.685013    5085 generic.go:356] "Generic (PLEG): container finished" podID="c7918962-8970-4916-960b-a31341090029" containerID="aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0" exitCode=0
Oct 30 20:14:42.685223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.685060    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"
Oct 30 20:14:42.685223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.685093    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" event={"ID":"c7918962-8970-4916-960b-a31341090029","Type":"ContainerDied","Data":"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"}
Oct 30 20:14:42.685223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.685152    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g" event={"ID":"c7918962-8970-4916-960b-a31341090029","Type":"ContainerDied","Data":"abf3e471f08d01f9f315e1e07aa0c0c0cfce3eb11ab93d8b7f3384178931e16d"}
Oct 30 20:14:42.685223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.685186    5085 scope.go:122] "RemoveContainer" containerID="aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"
Oct 30 20:14:42.686714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.686689    5085 generic.go:356] "Generic (PLEG): container finished" podID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerID="c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb" exitCode=0
Oct 30 20:14:42.686844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.686739    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" event={"ID":"ec296583-5915-4c7a-8ea7-96d3a7f6647d","Type":"ContainerDied","Data":"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"}
Oct 30 20:14:42.686844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.686777    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb" event={"ID":"ec296583-5915-4c7a-8ea7-96d3a7f6647d","Type":"ContainerDied","Data":"0f0fd563843d3fd0c1bc3f1668c7f09fd543116e8b44de7e94242390af7710e9"}
Oct 30 20:14:42.686844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.686741    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"
Oct 30 20:14:42.698227 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.698208    5085 scope.go:122] "RemoveContainer" containerID="aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"
Oct 30 20:14:42.698680 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:42.698636    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0\": container with ID starting with aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0 not found: ID does not exist" containerID="aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"
Oct 30 20:14:42.698795 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.698675    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0"} err="failed to get container status \"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0\": rpc error: code = NotFound desc = could not find container \"aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0\": container with ID starting with aaeb157f6b2efe1661e0aaf5b3d003772339eeb542d89313b7d96e468105e2e0 not found: ID does not exist"
Oct 30 20:14:42.698795 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.698698    5085 scope.go:122] "RemoveContainer" containerID="c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"
Oct 30 20:14:42.708049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.708029    5085 scope.go:122] "RemoveContainer" containerID="c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"
Oct 30 20:14:42.708334 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:42.708317    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb\": container with ID starting with c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb not found: ID does not exist" containerID="c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"
Oct 30 20:14:42.708402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.708341    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb"} err="failed to get container status \"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb\": rpc error: code = NotFound desc = could not find container \"c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb\": container with ID starting with c2a7d3d33e4b42415151d3a6574229179adf51cf16afd6fa3daf79291a3defeb not found: ID does not exist"
Oct 30 20:14:42.843710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.843633    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:14:42.889985 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.889946    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-389c4e-856bd97979-gdrrb"]
Oct 30 20:14:42.960643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.960552    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:14:42.969218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:42.969194    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-389c4e-79f9d89976-m948g"]
Oct 30 20:14:43.880154 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:43.880118    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:14:44.630912 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:44.630882    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7918962-8970-4916-960b-a31341090029" path="/var/lib/kubelet/pods/c7918962-8970-4916-960b-a31341090029/volumes"
Oct 30 20:14:44.631293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:44.631279    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" path="/var/lib/kubelet/pods/ec296583-5915-4c7a-8ea7-96d3a7f6647d/volumes"
Oct 30 20:14:47.170162 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170127    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:14:47.170673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170659    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5408b87c-83a3-4b2f-bb6e-0eb34f792b36" containerName="revoke-keys"
Oct 30 20:14:47.170725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170674    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5408b87c-83a3-4b2f-bb6e-0eb34f792b36" containerName="revoke-keys"
Oct 30 20:14:47.170725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170691    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing"
Oct 30 20:14:47.170725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170697    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing"
Oct 30 20:14:47.170827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170725    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing"
Oct 30 20:14:47.170827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170731    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing"
Oct 30 20:14:47.170827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170807    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5408b87c-83a3-4b2f-bb6e-0eb34f792b36" containerName="revoke-keys"
Oct 30 20:14:47.170827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170816    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7918962-8970-4916-960b-a31341090029" containerName="payload-processing"
Oct 30 20:14:47.170827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.170828    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec296583-5915-4c7a-8ea7-96d3a7f6647d" containerName="payload-pre-processing"
Oct 30 20:14:47.177437 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.177417    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.232753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.232723    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-98a19b-openshift-default-dockercfg-rhphv\""
Oct 30 20:14:47.282650 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.282614    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:14:47.295321 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295290    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295321 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295332    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295444    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295491    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295618    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295640    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxbl\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295732    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295761    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.295922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.295804    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397331    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397372    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397411    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397449    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397489    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397517    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxbl\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397623    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397661    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397729    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.397880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397813    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.398163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.397982    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.398163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.398058    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.398276 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.398206    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.398276 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.398237    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.399935 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.399909    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.400159 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.400139    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.456344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.456284    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.456497 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.456477    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxbl\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.492500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.492466    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:47.916906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.916881    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:14:47.918666 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:14:47.918638    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea6b2e4_fbe6_47a8_a87d_e97bd8a39017.slice/crio-efe9e9366bc67f844a3094ed177051c6eab0ab8683f83ee419611be388aedbc4 WatchSource:0}: Error finding container efe9e9366bc67f844a3094ed177051c6eab0ab8683f83ee419611be388aedbc4: Status 404 returned error can't find the container with id efe9e9366bc67f844a3094ed177051c6eab0ab8683f83ee419611be388aedbc4
Oct 30 20:14:47.921100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.921068    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:14:47.921174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.921133    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:14:47.921174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:47.921160    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:14:48.736115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:48.736061    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" event={"ID":"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017","Type":"ContainerStarted","Data":"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"}
Oct 30 20:14:48.736534 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:48.736153    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" event={"ID":"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017","Type":"ContainerStarted","Data":"efe9e9366bc67f844a3094ed177051c6eab0ab8683f83ee419611be388aedbc4"}
Oct 30 20:14:48.901872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:48.901816    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podStartSLOduration=1.901801761 podStartE2EDuration="1.901801761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:48.898228036 +0000 UTC m=+2998.402631782" watchObservedRunningTime="2026-10-30 20:14:48.901801761 +0000 UTC m=+2998.406205500"
Oct 30 20:14:49.492622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:49.492591    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:49.494401 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:49.494374    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:49.494531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:49.494424    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:50.493699 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:50.493666    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:50.494100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:50.493717    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:51.346408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:51.346353    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:14:51.351922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:51.351903    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:14:51.493223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:51.493178    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:51.493409 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:51.493245    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:51.939045 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:14:51.939012    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:14:52.492987 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:52.492952    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:52.493279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:52.493253    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:53.492985 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:53.492925    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:53.493392 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:53.493037    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:54.493455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:54.493417    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Oct 30 20:14:54.493849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:54.493494    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Oct 30 20:14:55.496848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.496818    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:55.497300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.497279    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:55.497892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.497867    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:14:55.605874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.605841    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:14:55.611155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.611133    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.706517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.706484    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d5d3ba-serving-cert\""
Oct 30 20:14:55.712613 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.712585    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:14:55.718644 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.718615    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc7g\" (UniqueName: \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.718789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.718662    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.718789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.718743    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.718789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.718765    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.820509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.820420    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc7g\" (UniqueName: \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.820509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.820478    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.820806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.820554    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.820806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.820594    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.821371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.821342    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.823252 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.823228    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.823448 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.823428    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.879285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.879246    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc7g\" (UniqueName: \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\") pod \"maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:55.925740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:55.925714    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:56.309056 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:56.309027    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:14:56.310098 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:14:56.310071    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846bcf37_02fe_4ead_b4c4_2dbc2b08ab1a.slice/crio-df3949f12aa7835b365db7d44bbe883a63a67836fe319b966d2ea17bb52e9e52 WatchSource:0}: Error finding container df3949f12aa7835b365db7d44bbe883a63a67836fe319b966d2ea17bb52e9e52: Status 404 returned error can't find the container with id df3949f12aa7835b365db7d44bbe883a63a67836fe319b966d2ea17bb52e9e52
Oct 30 20:14:56.810411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:56.810375    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" event={"ID":"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a","Type":"ContainerStarted","Data":"df3949f12aa7835b365db7d44bbe883a63a67836fe319b966d2ea17bb52e9e52"}
Oct 30 20:14:57.818849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:57.818762    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" event={"ID":"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a","Type":"ContainerStarted","Data":"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"}
Oct 30 20:14:57.819254 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:57.818853    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:14:57.952442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:14:57.952392    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" podStartSLOduration=2.023528552 podStartE2EDuration="2.95237595s" totalImagesPullingTime="928.847398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:57.951149327 +0000 UTC m=+3007.455553097" watchObservedRunningTime="2026-10-30 20:14:57.95237595 +0000 UTC m=+3007.456779694"
Oct 30 20:15:00.162108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.162069    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:00.166509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.166485    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.189215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.189189    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:00.193180 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.193160    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:00.193320 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.193291    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.197953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.197929    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.232947 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.232922    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:15:00.261146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.261115    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:00.266186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266157    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqsv\" (UniqueName: \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.266354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266210    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g9cq\" (UniqueName: \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.266354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266260    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.266354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266310    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.266354 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266341    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn8fc\" (UniqueName: \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.266605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.266367    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.312951 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.312922    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:00.325345 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.325079    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:00.367140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367107    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g9cq\" (UniqueName: \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.367316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367158    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.367316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367193    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.367316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367216    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8fc\" (UniqueName: \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.367316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367235    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.367543 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.367382    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqsv\" (UniqueName: \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.368022 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.368002    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.368089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.368061    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.368146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.368061    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.425997 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.425973    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8fc\" (UniqueName: \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\") pod \"maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.426145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.425973    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqsv\" (UniqueName: \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\") pod \"maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.426198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.426174    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g9cq\" (UniqueName: \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\") pod \"maas-api-key-cleanup-29889855-6wgpj\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.479684 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.479653    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:00.508306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.508279    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:00.515430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.515393    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:00.867943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.867858    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:00.904794 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:00.904763    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96c9006_5397_4504_98cf_8aef94c15bfd.slice/crio-92d7431292e05f2382f56a13511a19ea842e7e7b13f86e129011af7b9ff7b9dc WatchSource:0}: Error finding container 92d7431292e05f2382f56a13511a19ea842e7e7b13f86e129011af7b9ff7b9dc: Status 404 returned error can't find the container with id 92d7431292e05f2382f56a13511a19ea842e7e7b13f86e129011af7b9ff7b9dc
Oct 30 20:15:00.927769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.927743    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:00.967847 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:00.967659    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:00.970411 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:00.970382    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafea92ef_319e_496b_b49d_eabc415875c9.slice/crio-0e8057da033b6c71e57967fc093bd1c8cf31c0ef2ea6d03c17518bf3f77a484b WatchSource:0}: Error finding container 0e8057da033b6c71e57967fc093bd1c8cf31c0ef2ea6d03c17518bf3f77a484b: Status 404 returned error can't find the container with id 0e8057da033b6c71e57967fc093bd1c8cf31c0ef2ea6d03c17518bf3f77a484b
Oct 30 20:15:01.846325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.846292    5085 generic.go:356] "Generic (PLEG): container finished" podID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerID="77054c6645642dc25e593e36cbd42fb577226a25126980f4dcbd530de26fabdd" exitCode=60
Oct 30 20:15:01.846768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.846382    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" event={"ID":"e96c9006-5397-4504-98cf-8aef94c15bfd","Type":"ContainerDied","Data":"77054c6645642dc25e593e36cbd42fb577226a25126980f4dcbd530de26fabdd"}
Oct 30 20:15:01.846768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.846448    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" event={"ID":"e96c9006-5397-4504-98cf-8aef94c15bfd","Type":"ContainerStarted","Data":"92d7431292e05f2382f56a13511a19ea842e7e7b13f86e129011af7b9ff7b9dc"}
Oct 30 20:15:01.846768 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.846669    5085 scope.go:122] "RemoveContainer" containerID="77054c6645642dc25e593e36cbd42fb577226a25126980f4dcbd530de26fabdd"
Oct 30 20:15:01.848301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.848274    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerStarted","Data":"22f4df6c2392dcbe9696dfe63cfbbb7c6acf266f024cfffc5c1286115a86f92e"}
Oct 30 20:15:01.848362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.848319    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerStarted","Data":"0e8057da033b6c71e57967fc093bd1c8cf31c0ef2ea6d03c17518bf3f77a484b"}
Oct 30 20:15:01.850065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.850041    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" event={"ID":"a6529ce5-b813-4da2-955e-92067760759c","Type":"ContainerStarted","Data":"65c7f29d2c24d8a7f2c9c5e4aa414b6079e1a1e3bc2ca17afe685d0519e2f567"}
Oct 30 20:15:01.850299 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:01.850083    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" event={"ID":"a6529ce5-b813-4da2-955e-92067760759c","Type":"ContainerStarted","Data":"4546de2bf57fa61983117d8ff12a3fee2e4145700dc756ad9220ac446c578ad0"}
Oct 30 20:15:02.073689 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.073633    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" podStartSLOduration=2.073617932 podStartE2EDuration="2.073617932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.069896007 +0000 UTC m=+3011.574299754" watchObservedRunningTime="2026-10-30 20:15:02.073617932 +0000 UTC m=+3011.578021679"
Oct 30 20:15:02.180484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.180433    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" podStartSLOduration=2.180417604 podStartE2EDuration="2.180417604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.178916823 +0000 UTC m=+3011.683320569" watchObservedRunningTime="2026-10-30 20:15:02.180417604 +0000 UTC m=+3011.684821350"
Oct 30 20:15:02.858694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.858658    5085 generic.go:356] "Generic (PLEG): container finished" podID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerID="5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e" exitCode=60
Oct 30 20:15:02.859157 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.858758    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" event={"ID":"e96c9006-5397-4504-98cf-8aef94c15bfd","Type":"ContainerDied","Data":"5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e"}
Oct 30 20:15:02.859157 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.858815    5085 scope.go:122] "RemoveContainer" containerID="77054c6645642dc25e593e36cbd42fb577226a25126980f4dcbd530de26fabdd"
Oct 30 20:15:02.859157 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.859046    5085 scope.go:122] "RemoveContainer" containerID="5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e"
Oct 30 20:15:02.859361 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:02.859340    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-6wgpj_odh-ai-gateway-infra(e96c9006-5397-4504-98cf-8aef94c15bfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd"
Oct 30 20:15:02.860629 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.860605    5085 generic.go:356] "Generic (PLEG): container finished" podID="afea92ef-319e-496b-b49d-eabc415875c9" containerID="22f4df6c2392dcbe9696dfe63cfbbb7c6acf266f024cfffc5c1286115a86f92e" exitCode=60
Oct 30 20:15:02.860718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.860683    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerDied","Data":"22f4df6c2392dcbe9696dfe63cfbbb7c6acf266f024cfffc5c1286115a86f92e"}
Oct 30 20:15:02.860885 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.860868    5085 scope.go:122] "RemoveContainer" containerID="22f4df6c2392dcbe9696dfe63cfbbb7c6acf266f024cfffc5c1286115a86f92e"
Oct 30 20:15:02.862462 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.862443    5085 generic.go:356] "Generic (PLEG): container finished" podID="a6529ce5-b813-4da2-955e-92067760759c" containerID="65c7f29d2c24d8a7f2c9c5e4aa414b6079e1a1e3bc2ca17afe685d0519e2f567" exitCode=60
Oct 30 20:15:02.862534 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.862469    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" event={"ID":"a6529ce5-b813-4da2-955e-92067760759c","Type":"ContainerDied","Data":"65c7f29d2c24d8a7f2c9c5e4aa414b6079e1a1e3bc2ca17afe685d0519e2f567"}
Oct 30 20:15:02.862781 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:02.862764    5085 scope.go:122] "RemoveContainer" containerID="65c7f29d2c24d8a7f2c9c5e4aa414b6079e1a1e3bc2ca17afe685d0519e2f567"
Oct 30 20:15:03.830929 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.830898    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:15:03.870895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.870868    5085 generic.go:356] "Generic (PLEG): container finished" podID="a6529ce5-b813-4da2-955e-92067760759c" containerID="b479f3310ae68e3c2883561f920a243a6594976f51c136125b3c857c8cf1dfdc" exitCode=60
Oct 30 20:15:03.871234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.870947    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" event={"ID":"a6529ce5-b813-4da2-955e-92067760759c","Type":"ContainerDied","Data":"b479f3310ae68e3c2883561f920a243a6594976f51c136125b3c857c8cf1dfdc"}
Oct 30 20:15:03.871234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.871011    5085 scope.go:122] "RemoveContainer" containerID="65c7f29d2c24d8a7f2c9c5e4aa414b6079e1a1e3bc2ca17afe685d0519e2f567"
Oct 30 20:15:03.871340 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.871259    5085 scope.go:122] "RemoveContainer" containerID="b479f3310ae68e3c2883561f920a243a6594976f51c136125b3c857c8cf1dfdc"
Oct 30 20:15:03.871607 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:03.871579    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc_odh-ai-gateway-infra(a6529ce5-b813-4da2-955e-92067760759c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" podUID="a6529ce5-b813-4da2-955e-92067760759c"
Oct 30 20:15:03.872991 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.872975    5085 scope.go:122] "RemoveContainer" containerID="5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e"
Oct 30 20:15:03.873220 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:03.873202    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-6wgpj_odh-ai-gateway-infra(e96c9006-5397-4504-98cf-8aef94c15bfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd"
Oct 30 20:15:03.874526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.874507    5085 generic.go:356] "Generic (PLEG): container finished" podID="afea92ef-319e-496b-b49d-eabc415875c9" containerID="7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c" exitCode=60
Oct 30 20:15:03.874654 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.874622    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerDied","Data":"7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c"}
Oct 30 20:15:03.874967 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.874945    5085 scope.go:122] "RemoveContainer" containerID="7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c"
Oct 30 20:15:03.875246 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:03.875226    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m_odh-ai-gateway-infra(afea92ef-319e-496b-b49d-eabc415875c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" podUID="afea92ef-319e-496b-b49d-eabc415875c9"
Oct 30 20:15:03.929915 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:03.929896    5085 scope.go:122] "RemoveContainer" containerID="22f4df6c2392dcbe9696dfe63cfbbb7c6acf266f024cfffc5c1286115a86f92e"
Oct 30 20:15:04.517646 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:04.517604    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:15:04.517981 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:04.517949    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" podUID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" containerName="maas-api" containerID="cri-o://75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a" gracePeriod=30
Oct 30 20:15:04.883339 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:04.883244    5085 scope.go:122] "RemoveContainer" containerID="b479f3310ae68e3c2883561f920a243a6594976f51c136125b3c857c8cf1dfdc"
Oct 30 20:15:04.883833 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:04.883601    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc_odh-ai-gateway-infra(a6529ce5-b813-4da2-955e-92067760759c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" podUID="a6529ce5-b813-4da2-955e-92067760759c"
Oct 30 20:15:04.885062 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:04.885043    5085 scope.go:122] "RemoveContainer" containerID="7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c"
Oct 30 20:15:04.885278 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:04.885262    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m_odh-ai-gateway-infra(afea92ef-319e-496b-b49d-eabc415875c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" podUID="afea92ef-319e-496b-b49d-eabc415875c9"
Oct 30 20:15:05.633149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.633104    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:15:05.633423 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.633399    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" podUID="46ffb117-2577-4e3e-a6a3-c386582f791e" containerName="maas-api" containerID="cri-o://3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0" gracePeriod=30
Oct 30 20:15:05.809002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.808974    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:05.809976 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.809959    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:15:05.846216 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.846077    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") pod \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") "
Oct 30 20:15:05.849364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.848679    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\" (UniqueName: \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\") pod \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") "
Oct 30 20:15:05.849364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.848750    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\") pod \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") "
Oct 30 20:15:05.849364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.848876    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\") pod \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\" (UID: \"4a237a18-63a5-4fa9-a420-fe9a1d49d586\") "
Oct 30 20:15:05.850735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.850707    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca" pod "4a237a18-63a5-4fa9-a420-fe9a1d49d586" (UID: "4a237a18-63a5-4fa9-a420-fe9a1d49d586"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:05.870528 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.870314    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs" pod "4a237a18-63a5-4fa9-a420-fe9a1d49d586" (UID: "4a237a18-63a5-4fa9-a420-fe9a1d49d586"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:05.870528 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.870343    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls" pod "4a237a18-63a5-4fa9-a420-fe9a1d49d586" (UID: "4a237a18-63a5-4fa9-a420-fe9a1d49d586"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:05.871736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.870782    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg" pod "4a237a18-63a5-4fa9-a420-fe9a1d49d586" (UID: "4a237a18-63a5-4fa9-a420-fe9a1d49d586"). InnerVolumeSpecName "kube-api-access-4vcmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:05.888667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.888639    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:15:05.892145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.892117    5085 generic.go:356] "Generic (PLEG): container finished" podID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" containerID="75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a" exitCode=0
Oct 30 20:15:05.892263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.892153    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"
Oct 30 20:15:05.892263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.892196    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" event={"ID":"4a237a18-63a5-4fa9-a420-fe9a1d49d586","Type":"ContainerDied","Data":"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"}
Oct 30 20:15:05.892263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.892244    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6" event={"ID":"4a237a18-63a5-4fa9-a420-fe9a1d49d586","Type":"ContainerDied","Data":"f285d3100ee83f43c3336b17dec0f6bcf594af7ff9a2acbf1aec492cde09acde"}
Oct 30 20:15:05.892429 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.892268    5085 scope.go:122] "RemoveContainer" containerID="75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"
Oct 30 20:15:05.893631 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.893610    5085 generic.go:356] "Generic (PLEG): container finished" podID="46ffb117-2577-4e3e-a6a3-c386582f791e" containerID="3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0" exitCode=0
Oct 30 20:15:05.893730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.893701    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"
Oct 30 20:15:05.893798 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.893731    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" event={"ID":"46ffb117-2577-4e3e-a6a3-c386582f791e","Type":"ContainerDied","Data":"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"}
Oct 30 20:15:05.893798 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.893773    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh" event={"ID":"46ffb117-2577-4e3e-a6a3-c386582f791e","Type":"ContainerDied","Data":"ec7e4d7c5bcf9eec9e0778cdef0c9564073402df17193d2dcaff6eec9e09f774"}
Oct 30 20:15:05.903493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.903476    5085 scope.go:122] "RemoveContainer" containerID="75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"
Oct 30 20:15:05.903801 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:05.903778    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a\": container with ID starting with 75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a not found: ID does not exist" containerID="75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"
Oct 30 20:15:05.903851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.903813    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a"} err="failed to get container status \"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a\": rpc error: code = NotFound desc = could not find container \"75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a\": container with ID starting with 75cca50ebe235d6af9319fa4351095d924a54c71567ee605a47acf9cad32c59a not found: ID does not exist"
Oct 30 20:15:05.903851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.903838    5085 scope.go:122] "RemoveContainer" containerID="3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"
Oct 30 20:15:05.913766 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.913746    5085 scope.go:122] "RemoveContainer" containerID="3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"
Oct 30 20:15:05.914058 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:05.914040    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0\": container with ID starting with 3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0 not found: ID does not exist" containerID="3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"
Oct 30 20:15:05.914110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.914068    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0"} err="failed to get container status \"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0\": rpc error: code = NotFound desc = could not find container \"3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0\": container with ID starting with 3fb2ca0c49849bdb66e20aa81528d86b73db60bee134e5fcb613345be0d50dd0 not found: ID does not exist"
Oct 30 20:15:05.950509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.950474    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\") pod \"46ffb117-2577-4e3e-a6a3-c386582f791e\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") "
Oct 30 20:15:05.950709 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.950631    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") pod \"46ffb117-2577-4e3e-a6a3-c386582f791e\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") "
Oct 30 20:15:05.950709 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.950688    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\" (UniqueName: \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\") pod \"46ffb117-2577-4e3e-a6a3-c386582f791e\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") "
Oct 30 20:15:05.950854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.950748    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\") pod \"46ffb117-2577-4e3e-a6a3-c386582f791e\" (UID: \"46ffb117-2577-4e3e-a6a3-c386582f791e\") "
Oct 30 20:15:05.950854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.950840    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca" pod "46ffb117-2577-4e3e-a6a3-c386582f791e" (UID: "46ffb117-2577-4e3e-a6a3-c386582f791e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:05.951220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.951197    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:05.951298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.951227    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vcmg\" (UniqueName: \"kubernetes.io/projected/4a237a18-63a5-4fa9-a420-fe9a1d49d586-kube-api-access-4vcmg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:05.951298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.951243    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a237a18-63a5-4fa9-a420-fe9a1d49d586-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:05.951298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.951258    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ffb117-2577-4e3e-a6a3-c386582f791e-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:05.951298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.951272    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a237a18-63a5-4fa9-a420-fe9a1d49d586-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:05.953040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.953017    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs" pod "46ffb117-2577-4e3e-a6a3-c386582f791e" (UID: "46ffb117-2577-4e3e-a6a3-c386582f791e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:05.953531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.953511    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls" pod "46ffb117-2577-4e3e-a6a3-c386582f791e" (UID: "46ffb117-2577-4e3e-a6a3-c386582f791e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:05.957600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:05.955440    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999" pod "46ffb117-2577-4e3e-a6a3-c386582f791e" (UID: "46ffb117-2577-4e3e-a6a3-c386582f791e"). InnerVolumeSpecName "kube-api-access-6v999". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:06.052223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.052190    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:06.052385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.052227    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v999\" (UniqueName: \"kubernetes.io/projected/46ffb117-2577-4e3e-a6a3-c386582f791e-kube-api-access-6v999\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:06.052385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.052245    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46ffb117-2577-4e3e-a6a3-c386582f791e-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:06.055818 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.055791    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:06.146649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.146557    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:15:06.152822 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.152791    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-389c4e-79ffbbc7d7-vq9v6"]
Oct 30 20:15:06.153157 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.153133    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\") pod \"a6529ce5-b813-4da2-955e-92067760759c\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") "
Oct 30 20:15:06.153279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.153259    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\" (UniqueName: \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\") pod \"a6529ce5-b813-4da2-955e-92067760759c\" (UID: \"a6529ce5-b813-4da2-955e-92067760759c\") "
Oct 30 20:15:06.153668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.153637    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca" pod "a6529ce5-b813-4da2-955e-92067760759c" (UID: "a6529ce5-b813-4da2-955e-92067760759c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:06.155437 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.155412    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv" pod "a6529ce5-b813-4da2-955e-92067760759c" (UID: "a6529ce5-b813-4da2-955e-92067760759c"). InnerVolumeSpecName "kube-api-access-bgqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:06.255038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.255010    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6529ce5-b813-4da2-955e-92067760759c-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:06.255038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.255037    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqsv\" (UniqueName: \"kubernetes.io/projected/a6529ce5-b813-4da2-955e-92067760759c-kube-api-access-bgqsv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:06.364525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.364490    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:15:06.368857 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.368826    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-04679c-c6496c87f-rx7vh"]
Oct 30 20:15:06.633048 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.633013    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ffb117-2577-4e3e-a6a3-c386582f791e" path="/var/lib/kubelet/pods/46ffb117-2577-4e3e-a6a3-c386582f791e/volumes"
Oct 30 20:15:06.633527 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.633510    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" path="/var/lib/kubelet/pods/4a237a18-63a5-4fa9-a420-fe9a1d49d586/volumes"
Oct 30 20:15:06.901209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.901130    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"
Oct 30 20:15:06.901209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.901170    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc" event={"ID":"a6529ce5-b813-4da2-955e-92067760759c","Type":"ContainerDied","Data":"4546de2bf57fa61983117d8ff12a3fee2e4145700dc756ad9220ac446c578ad0"}
Oct 30 20:15:06.901770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:06.901235    5085 scope.go:122] "RemoveContainer" containerID="b479f3310ae68e3c2883561f920a243a6594976f51c136125b3c857c8cf1dfdc"
Oct 30 20:15:07.057863 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:07.057829    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:07.064840 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:07.064814    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-04679c-29889855-rfffc"]
Oct 30 20:15:07.882910 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:07.882873    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"]
Oct 30 20:15:07.885699 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:07.885678    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-04679c-bm8gf"]
Oct 30 20:15:08.640161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:08.640125    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5408b87c-83a3-4b2f-bb6e-0eb34f792b36" path="/var/lib/kubelet/pods/5408b87c-83a3-4b2f-bb6e-0eb34f792b36/volumes"
Oct 30 20:15:08.640555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:08.640540    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6529ce5-b813-4da2-955e-92067760759c" path="/var/lib/kubelet/pods/a6529ce5-b813-4da2-955e-92067760759c/volumes"
Oct 30 20:15:10.985436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.985394    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"]
Oct 30 20:15:10.986177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986155    5085 cpu_manager.go:414] "containerMap: removing container" podUID="46ffb117-2577-4e3e-a6a3-c386582f791e" containerName="maas-api"
Oct 30 20:15:10.986177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986176    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ffb117-2577-4e3e-a6a3-c386582f791e" containerName="maas-api"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986194    5085 cpu_manager.go:414] "containerMap: removing container" podUID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" containerName="maas-api"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986201    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" containerName="maas-api"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986220    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986227    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986405    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a237a18-63a5-4fa9-a420-fe9a1d49d586" containerName="maas-api"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986421    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:10.986440 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986434    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:10.986812 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.986456    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ffb117-2577-4e3e-a6a3-c386582f791e" containerName="maas-api"
Oct 30 20:15:10.991404 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:10.991383    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.045738 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.045700    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:11.097410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.097378    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"]
Oct 30 20:15:11.112946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.112915    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.113117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.113012    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76p86\" (UniqueName: \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.113170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.113111    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.214546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.214505    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76p86\" (UniqueName: \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.214546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.214551    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.214776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.214684    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.215367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.215342    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.216925 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.216905    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.275670 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.275550    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76p86\" (UniqueName: \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\") pod \"maas-api-revoke-sub-a53b2440-dsqc7\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.306747 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.306706    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:11.687649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.687628    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"]
Oct 30 20:15:11.690488 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:11.690459    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb39bc542_fea9_4022_a035_9af4808863ff.slice/crio-d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390 WatchSource:0}: Error finding container d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390: Status 404 returned error can't find the container with id d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390
Oct 30 20:15:11.944439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:11.944298    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7" event={"ID":"b39bc542-fea9-4022-a035-9af4808863ff","Type":"ContainerStarted","Data":"d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390"}
Oct 30 20:15:12.953133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:12.953097    5085 generic.go:356] "Generic (PLEG): container finished" podID="b39bc542-fea9-4022-a035-9af4808863ff" containerID="1b8f49463511ebd2cc956b8a34bcf34c524d210832b07576fc2864993634e0c2" exitCode=0
Oct 30 20:15:12.953609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:12.953180    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7" event={"ID":"b39bc542-fea9-4022-a035-9af4808863ff","Type":"ContainerDied","Data":"1b8f49463511ebd2cc956b8a34bcf34c524d210832b07576fc2864993634e0c2"}
Oct 30 20:15:13.624091 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:13.624056    5085 scope.go:122] "RemoveContainer" containerID="5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e"
Oct 30 20:15:14.101257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.101233    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:14.272668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.272639    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\" (UniqueName: \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\") pod \"b39bc542-fea9-4022-a035-9af4808863ff\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") "
Oct 30 20:15:14.272853 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.272701    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\") pod \"b39bc542-fea9-4022-a035-9af4808863ff\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") "
Oct 30 20:15:14.272853 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.272795    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\") pod \"b39bc542-fea9-4022-a035-9af4808863ff\" (UID: \"b39bc542-fea9-4022-a035-9af4808863ff\") "
Oct 30 20:15:14.273063 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.273040    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca" pod "b39bc542-fea9-4022-a035-9af4808863ff" (UID: "b39bc542-fea9-4022-a035-9af4808863ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:14.273371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.273352    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.274764 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.274743    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86" pod "b39bc542-fea9-4022-a035-9af4808863ff" (UID: "b39bc542-fea9-4022-a035-9af4808863ff"). InnerVolumeSpecName "kube-api-access-76p86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:14.274925 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.274903    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca" pod "b39bc542-fea9-4022-a035-9af4808863ff" (UID: "b39bc542-fea9-4022-a035-9af4808863ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:14.374529 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.374497    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76p86\" (UniqueName: \"kubernetes.io/projected/b39bc542-fea9-4022-a035-9af4808863ff-kube-api-access-76p86\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.374529 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.374522    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39bc542-fea9-4022-a035-9af4808863ff-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:14.968016 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.967985    5085 generic.go:356] "Generic (PLEG): container finished" podID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerID="45452323f4575ea49418291e77dd8840b932a5f93df5fae7bf465efc72097c85" exitCode=60
Oct 30 20:15:14.968208 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.968067    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" event={"ID":"e96c9006-5397-4504-98cf-8aef94c15bfd","Type":"ContainerDied","Data":"45452323f4575ea49418291e77dd8840b932a5f93df5fae7bf465efc72097c85"}
Oct 30 20:15:14.968208 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.968134    5085 scope.go:122] "RemoveContainer" containerID="5c6a0a96c0ce801f6ad01af37f2c3c3765b5dd8212ed731e76e8546cedab898e"
Oct 30 20:15:14.968603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.968584    5085 scope.go:122] "RemoveContainer" containerID="45452323f4575ea49418291e77dd8840b932a5f93df5fae7bf465efc72097c85"
Oct 30 20:15:14.968896 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:14.968874    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-6wgpj_odh-ai-gateway-infra(e96c9006-5397-4504-98cf-8aef94c15bfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd"
Oct 30 20:15:14.969916 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.969896    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7" event={"ID":"b39bc542-fea9-4022-a035-9af4808863ff","Type":"ContainerDied","Data":"d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390"}
Oct 30 20:15:14.970023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.969927    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"
Oct 30 20:15:14.970023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:14.969929    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3680d10db34a081dddb68079f307d0d37f444c7ad7e5c8109eb0945f918e390"
Oct 30 20:15:16.089349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.089310    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:16.244142 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.244114    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:16.396788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.396711    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\" (UniqueName: \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\") pod \"e96c9006-5397-4504-98cf-8aef94c15bfd\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") "
Oct 30 20:15:16.397061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.397031    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\") pod \"e96c9006-5397-4504-98cf-8aef94c15bfd\" (UID: \"e96c9006-5397-4504-98cf-8aef94c15bfd\") "
Oct 30 20:15:16.397599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.397542    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca" pod "e96c9006-5397-4504-98cf-8aef94c15bfd" (UID: "e96c9006-5397-4504-98cf-8aef94c15bfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:16.399651 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.399628    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq" pod "e96c9006-5397-4504-98cf-8aef94c15bfd" (UID: "e96c9006-5397-4504-98cf-8aef94c15bfd"). InnerVolumeSpecName "kube-api-access-7g9cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:16.498367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.498336    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g9cq\" (UniqueName: \"kubernetes.io/projected/e96c9006-5397-4504-98cf-8aef94c15bfd-kube-api-access-7g9cq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.498367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.498363    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e96c9006-5397-4504-98cf-8aef94c15bfd-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:16.986384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.986356    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"
Oct 30 20:15:16.986591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.986396    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj" event={"ID":"e96c9006-5397-4504-98cf-8aef94c15bfd","Type":"ContainerDied","Data":"92d7431292e05f2382f56a13511a19ea842e7e7b13f86e129011af7b9ff7b9dc"}
Oct 30 20:15:16.986591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:16.986467    5085 scope.go:122] "RemoveContainer" containerID="45452323f4575ea49418291e77dd8840b932a5f93df5fae7bf465efc72097c85"
Oct 30 20:15:17.144417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:17.144387    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:17.147521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:17.147495    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-6wgpj"]
Oct 30 20:15:17.624243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:17.624211    5085 scope.go:122] "RemoveContainer" containerID="7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c"
Oct 30 20:15:17.996674 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:17.996639    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerStarted","Data":"e5a1aa83cec96d73ff7b43cc053569b8e58987996e2d5adf8c5bcd7c58059e9c"}
Oct 30 20:15:18.631441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:18.631402    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" path="/var/lib/kubelet/pods/e96c9006-5397-4504-98cf-8aef94c15bfd/volumes"
Oct 30 20:15:19.005197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:19.005165    5085 generic.go:356] "Generic (PLEG): container finished" podID="afea92ef-319e-496b-b49d-eabc415875c9" containerID="e5a1aa83cec96d73ff7b43cc053569b8e58987996e2d5adf8c5bcd7c58059e9c" exitCode=60
Oct 30 20:15:19.005378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:19.005241    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerDied","Data":"e5a1aa83cec96d73ff7b43cc053569b8e58987996e2d5adf8c5bcd7c58059e9c"}
Oct 30 20:15:19.005378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:19.005309    5085 scope.go:122] "RemoveContainer" containerID="7347792d5b80d060729971e4ae23d50b711423cd782a1f82b9093af2e5ced50c"
Oct 30 20:15:19.005744 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:19.005720    5085 scope.go:122] "RemoveContainer" containerID="e5a1aa83cec96d73ff7b43cc053569b8e58987996e2d5adf8c5bcd7c58059e9c"
Oct 30 20:15:19.006047 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:19.006028    5085 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m_odh-ai-gateway-infra(afea92ef-319e-496b-b49d-eabc415875c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" podUID="afea92ef-319e-496b-b49d-eabc415875c9"
Oct 30 20:15:19.117431 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:19.117391    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:20.162152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.162128    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:20.196123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196097    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"]
Oct 30 20:15:20.196647 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196632    5085 cpu_manager.go:414] "containerMap: removing container" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196648    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196664    5085 cpu_manager.go:414] "containerMap: removing container" podUID="b39bc542-fea9-4022-a035-9af4808863ff" containerName="revoke-keys"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196669    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39bc542-fea9-4022-a035-9af4808863ff" containerName="revoke-keys"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196679    5085 cpu_manager.go:414] "containerMap: removing container" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196683    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196691    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196695    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.196707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196710    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196715    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6529ce5-b813-4da2-955e-92067760759c" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196728    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196734    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196741    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196745    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196824    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196835    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39bc542-fea9-4022-a035-9af4808863ff" containerName="revoke-keys"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196841    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196847    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.197124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.196858    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:20.200181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.200164    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.240928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.240900    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkr4s\" (UniqueName: \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.241065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.240938    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.241136 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.241065    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.250310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.250288    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:20.254371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.254341    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"]
Oct 30 20:15:20.341717 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.341640    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\") pod \"afea92ef-319e-496b-b49d-eabc415875c9\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") "
Oct 30 20:15:20.341893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.341727    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\" (UniqueName: \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\") pod \"afea92ef-319e-496b-b49d-eabc415875c9\" (UID: \"afea92ef-319e-496b-b49d-eabc415875c9\") "
Oct 30 20:15:20.341970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.341893    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkr4s\" (UniqueName: \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.341970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.341937    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.342091 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.342003    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.342091 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.341935    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca" pod "afea92ef-319e-496b-b49d-eabc415875c9" (UID: "afea92ef-319e-496b-b49d-eabc415875c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:20.342258 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.342236    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afea92ef-319e-496b-b49d-eabc415875c9-maas-api-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.342660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.342637    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.343916 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.343897    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc" pod "afea92ef-319e-496b-b49d-eabc415875c9" (UID: "afea92ef-319e-496b-b49d-eabc415875c9"). InnerVolumeSpecName "kube-api-access-cn8fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:20.344149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.344132    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.400426 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.400390    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkr4s\" (UniqueName: \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\") pod \"maas-api-revoke-sub-33a8ab3a-njwjl\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.442868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.442822    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn8fc\" (UniqueName: \"kubernetes.io/projected/afea92ef-319e-496b-b49d-eabc415875c9-kube-api-access-cn8fc\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.514025 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.513995    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:20.887654 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:20.887627    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"]
Oct 30 20:15:20.889124 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:20.889092    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ec4cd7_d15d_4897_aa9d_97d21cf13465.slice/crio-621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984 WatchSource:0}: Error finding container 621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984: Status 404 returned error can't find the container with id 621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984
Oct 30 20:15:21.020810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.020777    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl" event={"ID":"e3ec4cd7-d15d-4897-aa9d-97d21cf13465","Type":"ContainerStarted","Data":"621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984"}
Oct 30 20:15:21.022186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.022167    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"
Oct 30 20:15:21.022313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.022161    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m" event={"ID":"afea92ef-319e-496b-b49d-eabc415875c9","Type":"ContainerDied","Data":"0e8057da033b6c71e57967fc093bd1c8cf31c0ef2ea6d03c17518bf3f77a484b"}
Oct 30 20:15:21.022313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.022294    5085 scope.go:122] "RemoveContainer" containerID="e5a1aa83cec96d73ff7b43cc053569b8e58987996e2d5adf8c5bcd7c58059e9c"
Oct 30 20:15:21.178907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.178881    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:21.182970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:21.182945    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b0a183-29889855-75b2m"]
Oct 30 20:15:21.940685 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:21.940651    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:15:22.032627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:22.032551    5085 generic.go:356] "Generic (PLEG): container finished" podID="e3ec4cd7-d15d-4897-aa9d-97d21cf13465" containerID="5b32b2e800eaf731bc5c76e9fe6f7c6855cced81ea21f7997d36b95f5d2ab2a8" exitCode=0
Oct 30 20:15:22.032790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:22.032640    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl" event={"ID":"e3ec4cd7-d15d-4897-aa9d-97d21cf13465","Type":"ContainerDied","Data":"5b32b2e800eaf731bc5c76e9fe6f7c6855cced81ea21f7997d36b95f5d2ab2a8"}
Oct 30 20:15:22.631198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:22.631162    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afea92ef-319e-496b-b49d-eabc415875c9" path="/var/lib/kubelet/pods/afea92ef-319e-496b-b49d-eabc415875c9/volumes"
Oct 30 20:15:23.190342 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.190318    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:23.273289 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.273258    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\") pod \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") "
Oct 30 20:15:23.273461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.273321    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\" (UniqueName: \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\") pod \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") "
Oct 30 20:15:23.273461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.273430    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\") pod \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\" (UID: \"e3ec4cd7-d15d-4897-aa9d-97d21cf13465\") "
Oct 30 20:15:23.273851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.273812    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca" pod "e3ec4cd7-d15d-4897-aa9d-97d21cf13465" (UID: "e3ec4cd7-d15d-4897-aa9d-97d21cf13465"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:23.275407 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.275384    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca" pod "e3ec4cd7-d15d-4897-aa9d-97d21cf13465" (UID: "e3ec4cd7-d15d-4897-aa9d-97d21cf13465"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:23.275518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.275462    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s" pod "e3ec4cd7-d15d-4897-aa9d-97d21cf13465" (UID: "e3ec4cd7-d15d-4897-aa9d-97d21cf13465"). InnerVolumeSpecName "kube-api-access-bkr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:23.375211 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.375137    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkr4s\" (UniqueName: \"kubernetes.io/projected/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-kube-api-access-bkr4s\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.375211 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.375163    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:23.375211 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:23.375173    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ec4cd7-d15d-4897-aa9d-97d21cf13465-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:24.048254 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:24.048225    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"
Oct 30 20:15:24.048665 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:24.048269    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl" event={"ID":"e3ec4cd7-d15d-4897-aa9d-97d21cf13465","Type":"ContainerDied","Data":"621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984"}
Oct 30 20:15:24.048665 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:24.048322    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621c368ad7ae36ef4123a5de990436687eefde34ee2006582dfd7bcbe7cea984"
Oct 30 20:15:35.570570 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.570518    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"]
Oct 30 20:15:35.571167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571150    5085 cpu_manager.go:414] "containerMap: removing container" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:35.571167 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571167    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:35.571309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571181    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e3ec4cd7-d15d-4897-aa9d-97d21cf13465" containerName="revoke-keys"
Oct 30 20:15:35.571309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571186    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ec4cd7-d15d-4897-aa9d-97d21cf13465" containerName="revoke-keys"
Oct 30 20:15:35.571309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571295    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ec4cd7-d15d-4897-aa9d-97d21cf13465" containerName="revoke-keys"
Oct 30 20:15:35.571309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571305    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96c9006-5397-4504-98cf-8aef94c15bfd" containerName="cleanup"
Oct 30 20:15:35.571437 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.571318    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="afea92ef-319e-496b-b49d-eabc415875c9" containerName="cleanup"
Oct 30 20:15:35.576043 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.576026    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.628917 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.628887    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"]
Oct 30 20:15:35.672278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.672248    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:35.672508 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.672278    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:15:35.719937 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.719901    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.720070 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.719946    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.720134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.720098    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gf85\" (UniqueName: \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.821424 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.821333    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.821424 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.821391    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.821709 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.821515    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gf85\" (UniqueName: \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.822024 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.822002    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.823763 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.823735    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.880244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.880206    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gf85\" (UniqueName: \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\") pod \"maas-api-revoke-sub-61d18d9a-5zq7w\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:35.888756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:35.888731    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:36.264093 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:36.264068    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"]
Oct 30 20:15:36.265472 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:36.265445    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a2ad73f_9b90_4cd8_b37e_1d691856898a.slice/crio-ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f WatchSource:0}: Error finding container ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f: Status 404 returned error can't find the container with id ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f
Oct 30 20:15:37.132975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:37.132941    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w" event={"ID":"6a2ad73f-9b90-4cd8-b37e-1d691856898a","Type":"ContainerStarted","Data":"c1d353b4a2864d24e0af7b78f8a208472b90d4eee87bf411d9ac308cecb13e50"}
Oct 30 20:15:37.133408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:37.132993    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w" event={"ID":"6a2ad73f-9b90-4cd8-b37e-1d691856898a","Type":"ContainerStarted","Data":"ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f"}
Oct 30 20:15:37.245098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:37.245044    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w" podStartSLOduration=2.245027139 podStartE2EDuration="2.245027139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:37.243316646 +0000 UTC m=+3046.747720391" watchObservedRunningTime="2026-10-30 20:15:37.245027139 +0000 UTC m=+3046.749430928"
Oct 30 20:15:38.140338 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:38.140304    5085 generic.go:356] "Generic (PLEG): container finished" podID="6a2ad73f-9b90-4cd8-b37e-1d691856898a" containerID="c1d353b4a2864d24e0af7b78f8a208472b90d4eee87bf411d9ac308cecb13e50" exitCode=0
Oct 30 20:15:38.140744 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:38.140378    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w" event={"ID":"6a2ad73f-9b90-4cd8-b37e-1d691856898a","Type":"ContainerDied","Data":"c1d353b4a2864d24e0af7b78f8a208472b90d4eee87bf411d9ac308cecb13e50"}
Oct 30 20:15:39.302182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.302157    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:39.461662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.461616    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\" (UniqueName: \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\") pod \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") "
Oct 30 20:15:39.461883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.461684    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\") pod \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") "
Oct 30 20:15:39.461883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.461766    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\") pod \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\" (UID: \"6a2ad73f-9b90-4cd8-b37e-1d691856898a\") "
Oct 30 20:15:39.462135 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.462113    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca" pod "6a2ad73f-9b90-4cd8-b37e-1d691856898a" (UID: "6a2ad73f-9b90-4cd8-b37e-1d691856898a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:39.463701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.463674    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85" pod "6a2ad73f-9b90-4cd8-b37e-1d691856898a" (UID: "6a2ad73f-9b90-4cd8-b37e-1d691856898a"). InnerVolumeSpecName "kube-api-access-5gf85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:39.463809 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.463730    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca" pod "6a2ad73f-9b90-4cd8-b37e-1d691856898a" (UID: "6a2ad73f-9b90-4cd8-b37e-1d691856898a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:39.563212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.563178    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gf85\" (UniqueName: \"kubernetes.io/projected/6a2ad73f-9b90-4cd8-b37e-1d691856898a-kube-api-access-5gf85\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:39.563212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.563207    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:39.563212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:39.563218    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a2ad73f-9b90-4cd8-b37e-1d691856898a-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:40.154304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:40.154276    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"
Oct 30 20:15:40.154464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:40.154313    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w" event={"ID":"6a2ad73f-9b90-4cd8-b37e-1d691856898a","Type":"ContainerDied","Data":"ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f"}
Oct 30 20:15:40.154464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:40.154358    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab03a20f6a3259eb00ee2752fd47f2602aeccd9a2041adf88a0857f2c2acaf6f"
Oct 30 20:15:43.122483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:43.122445    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:15:44.174635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.174594    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:15:44.175271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.175125    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6a2ad73f-9b90-4cd8-b37e-1d691856898a" containerName="revoke-keys"
Oct 30 20:15:44.175271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.175137    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a2ad73f-9b90-4cd8-b37e-1d691856898a" containerName="revoke-keys"
Oct 30 20:15:44.175271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.175260    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a2ad73f-9b90-4cd8-b37e-1d691856898a" containerName="revoke-keys"
Oct 30 20:15:44.178553 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.178528    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.231122 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.231091    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-0b9215-openshift-default-dockercfg-4m7s5\""
Oct 30 20:15:44.283300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.283264    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:15:44.320773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.320740    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.322416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.321124    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.322777 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.322752    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.323082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.323062    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.323452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.323382    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.323639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.323622    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.323849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.323828    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kjq\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.324127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.324069    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.324410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.324389    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426211    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426290    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426324    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426360    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426396    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426438    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426453    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426509    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kjq\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.426711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.426607    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.428533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.427994    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.428738 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.428589    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.428887 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.428857    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.429140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.429114    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.430078 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.430043    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.431173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.431154    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.431939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.431919    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.485930 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.485894    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.488788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.488757    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kjq\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\") pod \"e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") " pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.496810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.496780    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:44.931017 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.930987    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:15:44.932306 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:44.932274    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab56326_d5ac_497b_a4f5_b9b5af4f4d38.slice/crio-b55fed1415a3d7ff92acfd42432b62d093ea7b6ec9c5ea8da0e72cbcf014afa6 WatchSource:0}: Error finding container b55fed1415a3d7ff92acfd42432b62d093ea7b6ec9c5ea8da0e72cbcf014afa6: Status 404 returned error can't find the container with id b55fed1415a3d7ff92acfd42432b62d093ea7b6ec9c5ea8da0e72cbcf014afa6
Oct 30 20:15:44.934923 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.934893    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:15:44.934989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.934953    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:15:44.934989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:44.934981    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:15:45.195476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:45.195380    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerStarted","Data":"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"}
Oct 30 20:15:45.195887 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:45.195483    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerStarted","Data":"b55fed1415a3d7ff92acfd42432b62d093ea7b6ec9c5ea8da0e72cbcf014afa6"}
Oct 30 20:15:45.312212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:45.312150    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podStartSLOduration=1.312135553 podStartE2EDuration="1.312135553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:45.310167509 +0000 UTC m=+3054.814571257" watchObservedRunningTime="2026-10-30 20:15:45.312135553 +0000 UTC m=+3054.816539299"
Oct 30 20:15:45.497991 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:45.497913    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:45.503271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:45.503247    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:46.202375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:46.202340    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:46.203278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:46.203252    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:15:47.270535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.270501    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"]
Oct 30 20:15:47.274324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.274304    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.326659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.326630    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:47.360664 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.360625    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmszn\" (UniqueName: \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.360874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.360731    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.360874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.360792    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.373030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.373004    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:15:47.377486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.377467    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"]
Oct 30 20:15:47.462365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.462333    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmszn\" (UniqueName: \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.462365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.462369    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.462634 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.462469    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.463164 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.463141    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.464964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.464940    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.521261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.521194    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmszn\" (UniqueName: \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\") pod \"maas-api-revoke-sub-b06305b2-x2469\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.589287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.589260    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:47.967225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:47.967195    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"]
Oct 30 20:15:47.967736 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:47.967697    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2aab29_1908_4cae_b468_434f669aa4d7.slice/crio-f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10 WatchSource:0}: Error finding container f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10: Status 404 returned error can't find the container with id f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10
Oct 30 20:15:48.216340 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:48.216262    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469" event={"ID":"cc2aab29-1908-4cae-b468-434f669aa4d7","Type":"ContainerStarted","Data":"f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10"}
Oct 30 20:15:49.224906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:49.224870    5085 generic.go:356] "Generic (PLEG): container finished" podID="cc2aab29-1908-4cae-b468-434f669aa4d7" containerID="ae9ebdbdd69d5c1794949e989417505a77d7d3fd301a2d8788470f2ef6020b88" exitCode=0
Oct 30 20:15:49.225352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:49.224913    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469" event={"ID":"cc2aab29-1908-4cae-b468-434f669aa4d7","Type":"ContainerDied","Data":"ae9ebdbdd69d5c1794949e989417505a77d7d3fd301a2d8788470f2ef6020b88"}
Oct 30 20:15:50.391797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.391770    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:50.498358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.498281    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\") pod \"cc2aab29-1908-4cae-b468-434f669aa4d7\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") "
Oct 30 20:15:50.498509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.498388    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\" (UniqueName: \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\") pod \"cc2aab29-1908-4cae-b468-434f669aa4d7\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") "
Oct 30 20:15:50.498587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.498531    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\") pod \"cc2aab29-1908-4cae-b468-434f669aa4d7\" (UID: \"cc2aab29-1908-4cae-b468-434f669aa4d7\") "
Oct 30 20:15:50.498850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.498826    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca" pod "cc2aab29-1908-4cae-b468-434f669aa4d7" (UID: "cc2aab29-1908-4cae-b468-434f669aa4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:50.499017 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.499000    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.500375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.500356    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca" pod "cc2aab29-1908-4cae-b468-434f669aa4d7" (UID: "cc2aab29-1908-4cae-b468-434f669aa4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:50.500518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.500499    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn" pod "cc2aab29-1908-4cae-b468-434f669aa4d7" (UID: "cc2aab29-1908-4cae-b468-434f669aa4d7"). InnerVolumeSpecName "kube-api-access-cmszn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:50.600206 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.600169    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc2aab29-1908-4cae-b468-434f669aa4d7-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:50.600206 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:50.600205    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmszn\" (UniqueName: \"kubernetes.io/projected/cc2aab29-1908-4cae-b468-434f669aa4d7-kube-api-access-cmszn\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:51.239359 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:51.239333    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"
Oct 30 20:15:51.239547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:51.239327    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469" event={"ID":"cc2aab29-1908-4cae-b468-434f669aa4d7","Type":"ContainerDied","Data":"f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10"}
Oct 30 20:15:51.239547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:51.239438    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89178c81d1def3f455d95f26c673d2bece397b853d46844248ea610ef48ed10"
Oct 30 20:15:51.939557 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:15:51.939516    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:15:53.017031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.016994    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"]
Oct 30 20:15:53.017909 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.017880    5085 cpu_manager.go:414] "containerMap: removing container" podUID="cc2aab29-1908-4cae-b468-434f669aa4d7" containerName="revoke-keys"
Oct 30 20:15:53.017909 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.017912    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc2aab29-1908-4cae-b468-434f669aa4d7" containerName="revoke-keys"
Oct 30 20:15:53.018106 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.018075    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc2aab29-1908-4cae-b468-434f669aa4d7" containerName="revoke-keys"
Oct 30 20:15:53.022300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.022279    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.074960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.074917    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"]
Oct 30 20:15:53.116556 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.116524    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:15:53.116718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.116600    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:15:53.129355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.129326    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.129478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.129389    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.129543 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.129489    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kx2x\" (UniqueName: \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.231030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.230986    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.231247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.231136    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.231247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.231180    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kx2x\" (UniqueName: \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.231991 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.231958    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.233526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.233492    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.290107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.290040    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kx2x\" (UniqueName: \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\") pod \"maas-api-revoke-sub-120790c2-84kmp\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.338577 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.338544    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:53.721895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:53.721865    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"]
Oct 30 20:15:53.723610 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:53.723576    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c55af63_bb1c_451b_929e_0f628245acdf.slice/crio-10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30 WatchSource:0}: Error finding container 10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30: Status 404 returned error can't find the container with id 10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30
Oct 30 20:15:54.260894 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:54.260794    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp" event={"ID":"6c55af63-bb1c-451b-929e-0f628245acdf","Type":"ContainerStarted","Data":"1453aa4a77e3d212c2ccff8ac44d9e9f0700f86f44ca0f0c1fef8e4110d26ba2"}
Oct 30 20:15:54.260894 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:54.260865    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp" event={"ID":"6c55af63-bb1c-451b-929e-0f628245acdf","Type":"ContainerStarted","Data":"10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30"}
Oct 30 20:15:55.270103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.270070    5085 generic.go:356] "Generic (PLEG): container finished" podID="6c55af63-bb1c-451b-929e-0f628245acdf" containerID="1453aa4a77e3d212c2ccff8ac44d9e9f0700f86f44ca0f0c1fef8e4110d26ba2" exitCode=0
Oct 30 20:15:55.270634 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.270163    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp" event={"ID":"6c55af63-bb1c-451b-929e-0f628245acdf","Type":"ContainerDied","Data":"1453aa4a77e3d212c2ccff8ac44d9e9f0700f86f44ca0f0c1fef8e4110d26ba2"}
Oct 30 20:15:55.868173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.868137    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:15:55.872381 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.872363    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:55.922742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.922715    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0b9215\""
Oct 30 20:15:55.923139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.923124    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0b9215-dockercfg-jf89v\""
Oct 30 20:15:55.935361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.935334    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:15:55.985763 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.985718    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftg9c\" (UniqueName: \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:55.986197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:55.986170    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.017897 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.017851    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:15:56.022183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.022160    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.072678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.072652    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:15:56.087549 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.087515    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.087710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.087662    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftg9c\" (UniqueName: \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.087820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.087798    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.087893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.087882    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7q7\" (UniqueName: \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.088572 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.088538    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.147110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.147047    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftg9c\" (UniqueName: \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\") pod \"payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.185805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.185768    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:56.189398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.188950    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7q7\" (UniqueName: \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.189398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.189025    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.189953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.189928    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.248203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.248147    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7q7\" (UniqueName: \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\") pod \"payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.336226 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.336192    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:56.425111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.419800    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:56.494108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.494081    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\") pod \"6c55af63-bb1c-451b-929e-0f628245acdf\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") "
Oct 30 20:15:56.494253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.494140    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\" (UniqueName: \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\") pod \"6c55af63-bb1c-451b-929e-0f628245acdf\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") "
Oct 30 20:15:56.494348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.494332    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\") pod \"6c55af63-bb1c-451b-929e-0f628245acdf\" (UID: \"6c55af63-bb1c-451b-929e-0f628245acdf\") "
Oct 30 20:15:56.494672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.494645    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca" pod "6c55af63-bb1c-451b-929e-0f628245acdf" (UID: "6c55af63-bb1c-451b-929e-0f628245acdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:56.494883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.494852    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:56.496038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.496017    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca" pod "6c55af63-bb1c-451b-929e-0f628245acdf" (UID: "6c55af63-bb1c-451b-929e-0f628245acdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:15:56.496149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.496132    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x" pod "6c55af63-bb1c-451b-929e-0f628245acdf" (UID: "6c55af63-bb1c-451b-929e-0f628245acdf"). InnerVolumeSpecName "kube-api-access-6kx2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:56.561232 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:56.561197    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e53c8f_de9a_4fd4_8181_f8f0b8582a76.slice/crio-5c1dae54d8861ef9211a9b1edbd514ca3022e24869fa0682fc8265e47beb1735 WatchSource:0}: Error finding container 5c1dae54d8861ef9211a9b1edbd514ca3022e24869fa0682fc8265e47beb1735: Status 404 returned error can't find the container with id 5c1dae54d8861ef9211a9b1edbd514ca3022e24869fa0682fc8265e47beb1735
Oct 30 20:15:56.562375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.562348    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:15:56.596113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.596080    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c55af63-bb1c-451b-929e-0f628245acdf-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:56.596113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.596109    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kx2x\" (UniqueName: \"kubernetes.io/projected/6c55af63-bb1c-451b-929e-0f628245acdf-kube-api-access-6kx2x\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:15:56.725884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:56.725863    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:15:56.726714 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:56.726686    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299e2a7c_3c07_4d23_8a78_78ab2a25b996.slice/crio-b0c47008ec33708494ac51fbdddb19dd346f590cd415845a33e0fb5c0253b9ad WatchSource:0}: Error finding container b0c47008ec33708494ac51fbdddb19dd346f590cd415845a33e0fb5c0253b9ad: Status 404 returned error can't find the container with id b0c47008ec33708494ac51fbdddb19dd346f590cd415845a33e0fb5c0253b9ad
Oct 30 20:15:57.103137 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.102966    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"]
Oct 30 20:15:57.103660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.103641    5085 cpu_manager.go:414] "containerMap: removing container" podUID="6c55af63-bb1c-451b-929e-0f628245acdf" containerName="revoke-keys"
Oct 30 20:15:57.103660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.103662    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c55af63-bb1c-451b-929e-0f628245acdf" containerName="revoke-keys"
Oct 30 20:15:57.103840 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.103774    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c55af63-bb1c-451b-929e-0f628245acdf" containerName="revoke-keys"
Oct 30 20:15:57.107358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.107337    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.165353 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.165318    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"]
Oct 30 20:15:57.204288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.204242    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq9z6\" (UniqueName: \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.204479 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.204320    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.204602 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.204558    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.288640 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.288599    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" event={"ID":"299e2a7c-3c07-4d23-8a78-78ab2a25b996","Type":"ContainerStarted","Data":"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"}
Oct 30 20:15:57.288822 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.288667    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" event={"ID":"299e2a7c-3c07-4d23-8a78-78ab2a25b996","Type":"ContainerStarted","Data":"b0c47008ec33708494ac51fbdddb19dd346f590cd415845a33e0fb5c0253b9ad"}
Oct 30 20:15:57.288822 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.288694    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:15:57.290293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.290264    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" event={"ID":"47e53c8f-de9a-4fd4-8181-f8f0b8582a76","Type":"ContainerStarted","Data":"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"}
Oct 30 20:15:57.290464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.290314    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" event={"ID":"47e53c8f-de9a-4fd4-8181-f8f0b8582a76","Type":"ContainerStarted","Data":"5c1dae54d8861ef9211a9b1edbd514ca3022e24869fa0682fc8265e47beb1735"}
Oct 30 20:15:57.290464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.290344    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:15:57.291680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.291659    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"
Oct 30 20:15:57.291799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.291705    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp" event={"ID":"6c55af63-bb1c-451b-929e-0f628245acdf","Type":"ContainerDied","Data":"10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30"}
Oct 30 20:15:57.291799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.291746    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10635ca56659477e7fc8a3319fb6506a69ca35f4c4029008af373fa7857a9c30"
Oct 30 20:15:57.305541 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.305507    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.305679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.305624    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq9z6\" (UniqueName: \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.305679 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.305670    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.306323 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.306306    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.307890 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.307874    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.373063 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.372979    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq9z6\" (UniqueName: \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\") pod \"maas-api-revoke-sub-1ff31619-zbtfc\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.401439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.401390    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" podStartSLOduration=2.401377017 podStartE2EDuration="2.401377017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:57.398604362 +0000 UTC m=+3066.903008106" watchObservedRunningTime="2026-10-30 20:15:57.401377017 +0000 UTC m=+3066.905780776"
Oct 30 20:15:57.420701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.420676    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:15:57.562197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.562131    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" podStartSLOduration=2.562110175 podStartE2EDuration="2.562110175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:57.559751197 +0000 UTC m=+3067.064154958" watchObservedRunningTime="2026-10-30 20:15:57.562110175 +0000 UTC m=+3067.066513924"
Oct 30 20:15:57.798494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:57.798462    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"]
Oct 30 20:15:57.802205 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:15:57.802173    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92a109a_47e5_4562_a344_63cc429f826d.slice/crio-ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e WatchSource:0}: Error finding container ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e: Status 404 returned error can't find the container with id ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e
Oct 30 20:15:58.076982 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:58.076901    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:15:58.077199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:58.077171    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy" containerID="cri-o://5d26f2b309189b8cfa2f2733136ccd7dd21103a190549da1a3c49ccdc2860dc6" gracePeriod=30
Oct 30 20:15:58.299624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:58.299591    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc" event={"ID":"d92a109a-47e5-4562-a344-63cc429f826d","Type":"ContainerStarted","Data":"289d8ac144f4a3a1ca9b2ee57589d8968026ecb943ddbcd385d9912b01ce9be2"}
Oct 30 20:15:58.299820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:58.299650    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc" event={"ID":"d92a109a-47e5-4562-a344-63cc429f826d","Type":"ContainerStarted","Data":"ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e"}
Oct 30 20:15:58.419590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:58.419478    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc" podStartSLOduration=1.419464154 podStartE2EDuration="1.419464154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:58.416063405 +0000 UTC m=+3067.920467153" watchObservedRunningTime="2026-10-30 20:15:58.419464154 +0000 UTC m=+3067.923867901"
Oct 30 20:15:59.308304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:59.308271    5085 generic.go:356] "Generic (PLEG): container finished" podID="d92a109a-47e5-4562-a344-63cc429f826d" containerID="289d8ac144f4a3a1ca9b2ee57589d8968026ecb943ddbcd385d9912b01ce9be2" exitCode=0
Oct 30 20:15:59.308494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:15:59.308329    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc" event={"ID":"d92a109a-47e5-4562-a344-63cc429f826d","Type":"ContainerDied","Data":"289d8ac144f4a3a1ca9b2ee57589d8968026ecb943ddbcd385d9912b01ce9be2"}
Oct 30 20:16:00.467870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.467848    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:16:00.546532 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.546503    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\" (UniqueName: \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\") pod \"d92a109a-47e5-4562-a344-63cc429f826d\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") "
Oct 30 20:16:00.546718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.546547    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\") pod \"d92a109a-47e5-4562-a344-63cc429f826d\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") "
Oct 30 20:16:00.546718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.546650    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\") pod \"d92a109a-47e5-4562-a344-63cc429f826d\" (UID: \"d92a109a-47e5-4562-a344-63cc429f826d\") "
Oct 30 20:16:00.547085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.547051    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca" pod "d92a109a-47e5-4562-a344-63cc429f826d" (UID: "d92a109a-47e5-4562-a344-63cc429f826d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:00.548753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.548731    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca" pod "d92a109a-47e5-4562-a344-63cc429f826d" (UID: "d92a109a-47e5-4562-a344-63cc429f826d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:00.548835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.548748    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6" pod "d92a109a-47e5-4562-a344-63cc429f826d" (UID: "d92a109a-47e5-4562-a344-63cc429f826d"). InnerVolumeSpecName "kube-api-access-pq9z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:00.648214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.648190    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq9z6\" (UniqueName: \"kubernetes.io/projected/d92a109a-47e5-4562-a344-63cc429f826d-kube-api-access-pq9z6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:00.648214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.648214    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:00.648373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:00.648224    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d92a109a-47e5-4562-a344-63cc429f826d-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:01.237898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.237861    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:16:01.238509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.238491    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d92a109a-47e5-4562-a344-63cc429f826d" containerName="revoke-keys"
Oct 30 20:16:01.238509 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.238509    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d92a109a-47e5-4562-a344-63cc429f826d" containerName="revoke-keys"
Oct 30 20:16:01.238711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.238616    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d92a109a-47e5-4562-a344-63cc429f826d" containerName="revoke-keys"
Oct 30 20:16:01.242304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.242284    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.292316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.292290    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bbfa07-openshift-default-dockercfg-vj7kd\""
Oct 30 20:16:01.325033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.325008    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"
Oct 30 20:16:01.325171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.325036    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc" event={"ID":"d92a109a-47e5-4562-a344-63cc429f826d","Type":"ContainerDied","Data":"ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e"}
Oct 30 20:16:01.325171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.325081    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba493249138f7cb0b8950a220c7e851a4d480552a402886ff1d00ca2e83d496e"
Oct 30 20:16:01.342733 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.342707    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:16:01.356946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.356914    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.356961    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357109    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357258 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357225    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357253    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77g8p\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357299    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357460    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357550 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357538    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.357628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.357581    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458522    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458592    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458617    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458658    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458685    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.458770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458731    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458845    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458895    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77g8p\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458923    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458938    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.458983    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.459117    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.459256    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.459474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.459439    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.461004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.460979    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.461221 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.461204    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.517177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.517100    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.517500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.517195    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77g8p\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\") pod \"e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") " pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.556973 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.556944    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:01.978384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.978358    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:16:01.979302 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:01.979277    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67737ccd_a64c_4995_aac6_d7f13ba3b93a.slice/crio-d3ec974c3b5af24a5e29117787060e0fca66f3a2db1bfa8e1dc07f530e4b368f WatchSource:0}: Error finding container d3ec974c3b5af24a5e29117787060e0fca66f3a2db1bfa8e1dc07f530e4b368f: Status 404 returned error can't find the container with id d3ec974c3b5af24a5e29117787060e0fca66f3a2db1bfa8e1dc07f530e4b368f
Oct 30 20:16:01.981736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.981702    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:01.981795 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.981768    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:01.981836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:01.981796    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:02.333421 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.333331    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" event={"ID":"67737ccd-a64c-4995-aac6-d7f13ba3b93a","Type":"ContainerStarted","Data":"a8dbf9c385776a7aada6f19ff6587905f8716d2fa6e9dccfcfed1b932448a3bc"}
Oct 30 20:16:02.333421 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.333393    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" event={"ID":"67737ccd-a64c-4995-aac6-d7f13ba3b93a","Type":"ContainerStarted","Data":"d3ec974c3b5af24a5e29117787060e0fca66f3a2db1bfa8e1dc07f530e4b368f"}
Oct 30 20:16:02.455558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.455505    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podStartSLOduration=1.455490101 podStartE2EDuration="1.455490101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:02.450926736 +0000 UTC m=+3071.955330484" watchObservedRunningTime="2026-10-30 20:16:02.455490101 +0000 UTC m=+3071.959893848"
Oct 30 20:16:02.526434 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.526397    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"]
Oct 30 20:16:02.530973 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.530952    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.563887 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.563849    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:02.581922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.581888    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:02.582143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.582014    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:16:02.601018 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.600931    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"]
Oct 30 20:16:02.681343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.681303    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.681852 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.681825    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smq6k\" (UniqueName: \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.681998 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.681880    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.783329 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.783288    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smq6k\" (UniqueName: \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.783520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.783358    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.783520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.783487    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.784874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.784842    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.786819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.786790    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.842306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.842279    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smq6k\" (UniqueName: \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\") pod \"maas-api-revoke-sub-b2f60440-mv9rt\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:02.845044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:02.845022    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:03.252308 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.252253    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"]
Oct 30 20:16:03.344117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.344088    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt" event={"ID":"d30fdcdb-b163-43d2-98d8-454ae19d6c71","Type":"ContainerStarted","Data":"ed434faefe7d3ebf55f25c727f0e2f1f796365abc2442a6b9bdb8a27f93dfb68"}
Oct 30 20:16:03.346064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.346039    5085 generic.go:356] "Generic (PLEG): container finished" podID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerID="5d26f2b309189b8cfa2f2733136ccd7dd21103a190549da1a3c49ccdc2860dc6" exitCode=0
Oct 30 20:16:03.346186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.346108    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerDied","Data":"5d26f2b309189b8cfa2f2733136ccd7dd21103a190549da1a3c49ccdc2860dc6"}
Oct 30 20:16:03.346186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.346160    5085 scope.go:122] "RemoveContainer" containerID="71073c27b1578f5628d9c418cdc05777a49c978e42b14a9245ccbb41a195fc34"
Oct 30 20:16:03.347317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.347300    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:16:03.492948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.492912    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493039    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493070    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493103    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493159    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493214    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493261    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493400 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493337    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.493514 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.493405    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\") pod \"cbac9a96-319e-4a4b-9542-6c387618c6f6\" (UID: \"cbac9a96-319e-4a4b-9542-6c387618c6f6\") "
Oct 30 20:16:03.496486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.495624    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.496486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.496247    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.496486 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.496443    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.497327 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.497302    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:03.497820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.497793    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:16:03.498051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.498003    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.504305 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.504277    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "kube-api-access-cmk46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.504410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.504354    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:16:03.504493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.504420    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token" pod "cbac9a96-319e-4a4b-9542-6c387618c6f6" (UID: "cbac9a96-319e-4a4b-9542-6c387618c6f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:03.581806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.581768    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:03.582202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.581829    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:03.594854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594825    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.594854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594850    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594867    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594877    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594885    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594894    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594902    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmk46\" (UniqueName: \"kubernetes.io/projected/cbac9a96-319e-4a4b-9542-6c387618c6f6-kube-api-access-cmk46\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594913    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbac9a96-319e-4a4b-9542-6c387618c6f6-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:03.595031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:03.594921    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbac9a96-319e-4a4b-9542-6c387618c6f6-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:04.354253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.354219    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt" event={"ID":"d30fdcdb-b163-43d2-98d8-454ae19d6c71","Type":"ContainerStarted","Data":"c5c4d825ef78e677a12c454a255a1a81c8682189655f1542b39e73606c27b283"}
Oct 30 20:16:04.355819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.355794    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"
Oct 30 20:16:04.355954 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.355843    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx" event={"ID":"cbac9a96-319e-4a4b-9542-6c387618c6f6","Type":"ContainerDied","Data":"89e4ddc3712491ab0111b138a5ae06256f6d6d5e80626842167c31e713807e2e"}
Oct 30 20:16:04.355954 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.355911    5085 scope.go:122] "RemoveContainer" containerID="5d26f2b309189b8cfa2f2733136ccd7dd21103a190549da1a3c49ccdc2860dc6"
Oct 30 20:16:04.578588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.566928    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:04.578588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.566990    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:04.613406 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.613303    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt" podStartSLOduration=2.613288993 podStartE2EDuration="2.613288993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:04.48188721 +0000 UTC m=+3073.986290982" watchObservedRunningTime="2026-10-30 20:16:04.613288993 +0000 UTC m=+3074.117692740"
Oct 30 20:16:04.685730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.685691    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:16:04.691103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:04.691037    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-389c4e-openshift-default-6c8ddcd78f-rnkwx"]
Oct 30 20:16:05.389432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:05.389396    5085 generic.go:356] "Generic (PLEG): container finished" podID="d30fdcdb-b163-43d2-98d8-454ae19d6c71" containerID="c5c4d825ef78e677a12c454a255a1a81c8682189655f1542b39e73606c27b283" exitCode=0
Oct 30 20:16:05.389612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:05.389444    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt" event={"ID":"d30fdcdb-b163-43d2-98d8-454ae19d6c71","Type":"ContainerDied","Data":"c5c4d825ef78e677a12c454a255a1a81c8682189655f1542b39e73606c27b283"}
Oct 30 20:16:05.565623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:05.564906    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:05.565623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:05.564965    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:06.572953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.572917    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:06.573490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.573459    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:06.589756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.589729    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:06.625892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.625869    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\") pod \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") "
Oct 30 20:16:06.626046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.625938    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\") pod \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") "
Oct 30 20:16:06.626046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.625980    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\" (UniqueName: \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\") pod \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\" (UID: \"d30fdcdb-b163-43d2-98d8-454ae19d6c71\") "
Oct 30 20:16:06.626253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.626227    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca" pod "d30fdcdb-b163-43d2-98d8-454ae19d6c71" (UID: "d30fdcdb-b163-43d2-98d8-454ae19d6c71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:06.628070 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.628047    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca" pod "d30fdcdb-b163-43d2-98d8-454ae19d6c71" (UID: "d30fdcdb-b163-43d2-98d8-454ae19d6c71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:06.628335 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.628320    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k" pod "d30fdcdb-b163-43d2-98d8-454ae19d6c71" (UID: "d30fdcdb-b163-43d2-98d8-454ae19d6c71"). InnerVolumeSpecName "kube-api-access-smq6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:06.634182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.634151    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" path="/var/lib/kubelet/pods/cbac9a96-319e-4a4b-9542-6c387618c6f6/volumes"
Oct 30 20:16:06.727337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.727220    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:06.727337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.727255    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30fdcdb-b163-43d2-98d8-454ae19d6c71-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:06.727337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:06.727271    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smq6k\" (UniqueName: \"kubernetes.io/projected/d30fdcdb-b163-43d2-98d8-454ae19d6c71-kube-api-access-smq6k\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:07.405466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.405435    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"
Oct 30 20:16:07.405653 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.405469    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt" event={"ID":"d30fdcdb-b163-43d2-98d8-454ae19d6c71","Type":"ContainerDied","Data":"ed434faefe7d3ebf55f25c727f0e2f1f796365abc2442a6b9bdb8a27f93dfb68"}
Oct 30 20:16:07.405653 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.405520    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed434faefe7d3ebf55f25c727f0e2f1f796365abc2442a6b9bdb8a27f93dfb68"
Oct 30 20:16:07.564588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.561232    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:07.564588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.561296    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:07.891751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.891709    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"]
Oct 30 20:16:07.892672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892583    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d30fdcdb-b163-43d2-98d8-454ae19d6c71" containerName="revoke-keys"
Oct 30 20:16:07.892672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892608    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30fdcdb-b163-43d2-98d8-454ae19d6c71" containerName="revoke-keys"
Oct 30 20:16:07.892672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892646    5085 cpu_manager.go:414] "containerMap: removing container" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.892672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892656    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.892874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892683    5085 cpu_manager.go:414] "containerMap: removing container" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.892874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892692    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.892948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892900    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30fdcdb-b163-43d2-98d8-454ae19d6c71" containerName="revoke-keys"
Oct 30 20:16:07.892948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892915    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.892948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.892927    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbac9a96-319e-4a4b-9542-6c387618c6f6" containerName="istio-proxy"
Oct 30 20:16:07.897218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.897197    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:07.943890 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.943860    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"]
Oct 30 20:16:07.970843 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.970805    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:07.978571 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:07.978528    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:16:08.043706 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.043675    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.043877 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.043833    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.043918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.043879    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmg5\" (UniqueName: \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.144625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.144534    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.144807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.144636    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmg5\" (UniqueName: \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.144807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.144730    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.145196 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.145168    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.147082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.147058    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.202781 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.202746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmg5\" (UniqueName: \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\") pod \"maas-api-revoke-sub-95d00d90-bnmhz\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.212609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.212585    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:08.302853 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.302822    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:16:08.303029 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.302934    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:16:08.557640 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.557604    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 30 20:16:08.557884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.557848    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 30 20:16:08.592606 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:08.592546    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ae12b7_75c0_481b_8c50_64f16b09d1c5.slice/crio-fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4 WatchSource:0}: Error finding container fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4: Status 404 returned error can't find the container with id fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4
Oct 30 20:16:08.593111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:08.593085    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"]
Oct 30 20:16:09.421121 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.421085    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz" event={"ID":"34ae12b7-75c0-481b-8c50-64f16b09d1c5","Type":"ContainerStarted","Data":"420b278a567ae28e4d18eb7329782d0095a9929d4b7a42c47548bfb25ebd24eb"}
Oct 30 20:16:09.421630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.421152    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz" event={"ID":"34ae12b7-75c0-481b-8c50-64f16b09d1c5","Type":"ContainerStarted","Data":"fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4"}
Oct 30 20:16:09.535040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.534978    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz" podStartSLOduration=2.534961658 podStartE2EDuration="2.534961658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:09.532528609 +0000 UTC m=+3079.036932355" watchObservedRunningTime="2026-10-30 20:16:09.534961658 +0000 UTC m=+3079.039365405"
Oct 30 20:16:09.566603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.566571    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:09.567067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.567047    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:09.567741 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:09.567719    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:16:10.431119 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:10.431079    5085 generic.go:356] "Generic (PLEG): container finished" podID="34ae12b7-75c0-481b-8c50-64f16b09d1c5" containerID="420b278a567ae28e4d18eb7329782d0095a9929d4b7a42c47548bfb25ebd24eb" exitCode=0
Oct 30 20:16:10.431627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:10.431183    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz" event={"ID":"34ae12b7-75c0-481b-8c50-64f16b09d1c5","Type":"ContainerDied","Data":"420b278a567ae28e4d18eb7329782d0095a9929d4b7a42c47548bfb25ebd24eb"}
Oct 30 20:16:11.790949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.790918    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:11.818294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.817454    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\") pod \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") "
Oct 30 20:16:11.818294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.817543    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\" (UniqueName: \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\") pod \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") "
Oct 30 20:16:11.818294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.817686    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\") pod \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\" (UID: \"34ae12b7-75c0-481b-8c50-64f16b09d1c5\") "
Oct 30 20:16:11.819076 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.818767    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca" pod "34ae12b7-75c0-481b-8c50-64f16b09d1c5" (UID: "34ae12b7-75c0-481b-8c50-64f16b09d1c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:11.823878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.823831    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca" pod "34ae12b7-75c0-481b-8c50-64f16b09d1c5" (UID: "34ae12b7-75c0-481b-8c50-64f16b09d1c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:11.824710 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.824683    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5" pod "34ae12b7-75c0-481b-8c50-64f16b09d1c5" (UID: "34ae12b7-75c0-481b-8c50-64f16b09d1c5"). InnerVolumeSpecName "kube-api-access-9dmg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:11.922586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.922429    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dmg5\" (UniqueName: \"kubernetes.io/projected/34ae12b7-75c0-481b-8c50-64f16b09d1c5-kube-api-access-9dmg5\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:11.922586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.922466    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:11.922586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:11.922485    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34ae12b7-75c0-481b-8c50-64f16b09d1c5-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:12.447156 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:12.447129    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"
Oct 30 20:16:12.447334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:12.447158    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz" event={"ID":"34ae12b7-75c0-481b-8c50-64f16b09d1c5","Type":"ContainerDied","Data":"fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4"}
Oct 30 20:16:12.447334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:12.447205    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc5ef5b54a6acb09b441b5a6c75d246754878a97e894c4085878fcb2d2d34b4"
Oct 30 20:16:12.606740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:12.606703    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:16:12.712648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:12.712553    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:16:16.992779 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:16.992743    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:16:16.993300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:16.993286    5085 cpu_manager.go:414] "containerMap: removing container" podUID="34ae12b7-75c0-481b-8c50-64f16b09d1c5" containerName="revoke-keys"
Oct 30 20:16:16.993347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:16.993300    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ae12b7-75c0-481b-8c50-64f16b09d1c5" containerName="revoke-keys"
Oct 30 20:16:16.993442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:16.993433    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ae12b7-75c0-481b-8c50-64f16b09d1c5" containerName="revoke-keys"
Oct 30 20:16:16.996949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:16.996928    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.080031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.079984    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4x7\" (UniqueName: \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.080317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.080291    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.104735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.104699    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-547082\""
Oct 30 20:16:17.108355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.105034    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-547082-dockercfg-q4kt7\""
Oct 30 20:16:17.109587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.109547    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:16:17.149454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.149422    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:16:17.153502 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.153476    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.181558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.181526    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmt8\" (UniqueName: \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.181724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.181676    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4x7\" (UniqueName: \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.181724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.181699    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.181724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.181722    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.182376 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.182355    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.211082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.211049    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:16:17.243641 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.243556    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4x7\" (UniqueName: \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\") pod \"payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.283086 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.283052    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmt8\" (UniqueName: \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.283255 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.283161    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.283938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.283918    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.310954 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.310919    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:17.342174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.342135    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmt8\" (UniqueName: \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\") pod \"payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.468880 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.468848    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:17.607177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.607109    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" podUID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" containerName="payload-processing" containerID="cri-o://6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c" gracePeriod=25
Oct 30 20:16:17.687650 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.687620    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:16:17.687833 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:17.687673    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad9cec6_091a_48a4_afbf_2c86542c0df2.slice/crio-8499021afb44710fe9d6e45d91f4930ee40031d9a06d254e3bc432d1fed084d6 WatchSource:0}: Error finding container 8499021afb44710fe9d6e45d91f4930ee40031d9a06d254e3bc432d1fed084d6: Status 404 returned error can't find the container with id 8499021afb44710fe9d6e45d91f4930ee40031d9a06d254e3bc432d1fed084d6
Oct 30 20:16:17.712807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.712779    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" podUID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" containerName="payload-pre-processing" containerID="cri-o://1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e" gracePeriod=25
Oct 30 20:16:17.844394 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.843825    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:16:17.846981 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:17.846949    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95422b84_7885_4e95_b7c9_88dd2344bfea.slice/crio-0765c3932fa1e113b6465fa03e03aa55cd578e0d6fcb35c7d976cf068b6db12a WatchSource:0}: Error finding container 0765c3932fa1e113b6465fa03e03aa55cd578e0d6fcb35c7d976cf068b6db12a: Status 404 returned error can't find the container with id 0765c3932fa1e113b6465fa03e03aa55cd578e0d6fcb35c7d976cf068b6db12a
Oct 30 20:16:17.868472 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.868446    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:16:17.894105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.892132    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\" (UniqueName: \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\") pod \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") "
Oct 30 20:16:17.894105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.892194    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\") pod \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\" (UID: \"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa\") "
Oct 30 20:16:17.894105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.892817    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume" pod "7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" (UID: "7d5c323e-bfc2-47a8-85fc-ef9a96537aaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:17.895033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.895013    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5" pod "7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" (UID: "7d5c323e-bfc2-47a8-85fc-ef9a96537aaa"). InnerVolumeSpecName "kube-api-access-cnbb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:17.987425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.987397    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:16:17.993636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.993607    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbb5\" (UniqueName: \"kubernetes.io/projected/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-kube-api-access-cnbb5\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:17.993957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:17.993639    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.094938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.094846    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\" (UniqueName: \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\") pod \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") "
Oct 30 20:16:18.095126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.095073    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\") pod \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\" (UID: \"f996f1e3-ae0e-4f90-9e93-86ce7e747db0\") "
Oct 30 20:16:18.095457 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.095433    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume" pod "f996f1e3-ae0e-4f90-9e93-86ce7e747db0" (UID: "f996f1e3-ae0e-4f90-9e93-86ce7e747db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:18.096987 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.096966    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm" pod "f996f1e3-ae0e-4f90-9e93-86ce7e747db0" (UID: "f996f1e3-ae0e-4f90-9e93-86ce7e747db0"). InnerVolumeSpecName "kube-api-access-zqhjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:18.196272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.196238    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.196272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.196266    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqhjm\" (UniqueName: \"kubernetes.io/projected/f996f1e3-ae0e-4f90-9e93-86ce7e747db0-kube-api-access-zqhjm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:18.492643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.492610    5085 generic.go:356] "Generic (PLEG): container finished" podID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" containerID="1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e" exitCode=0
Oct 30 20:16:18.492830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.492671    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"
Oct 30 20:16:18.492830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.492694    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" event={"ID":"f996f1e3-ae0e-4f90-9e93-86ce7e747db0","Type":"ContainerDied","Data":"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"}
Oct 30 20:16:18.492830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.492783    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj" event={"ID":"f996f1e3-ae0e-4f90-9e93-86ce7e747db0","Type":"ContainerDied","Data":"9899e6bc4e3ee3271a7e00b6240e85fc4b0bb2bbe6740fa72e781f4d675e2af2"}
Oct 30 20:16:18.492830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.492816    5085 scope.go:122] "RemoveContainer" containerID="1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"
Oct 30 20:16:18.494228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.494208    5085 generic.go:356] "Generic (PLEG): container finished" podID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" containerID="6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c" exitCode=0
Oct 30 20:16:18.494333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.494314    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" event={"ID":"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa","Type":"ContainerDied","Data":"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"}
Oct 30 20:16:18.494399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.494355    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x" event={"ID":"7d5c323e-bfc2-47a8-85fc-ef9a96537aaa","Type":"ContainerDied","Data":"07768000bd5c922b56b47d07f0957fdd9d718f9c60bcf701152db7000a3921ff"}
Oct 30 20:16:18.494399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.494325    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"
Oct 30 20:16:18.496340 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.496313    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" event={"ID":"95422b84-7885-4e95-b7c9-88dd2344bfea","Type":"ContainerStarted","Data":"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"}
Oct 30 20:16:18.496521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.496499    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" event={"ID":"95422b84-7885-4e95-b7c9-88dd2344bfea","Type":"ContainerStarted","Data":"0765c3932fa1e113b6465fa03e03aa55cd578e0d6fcb35c7d976cf068b6db12a"}
Oct 30 20:16:18.496750 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.496685    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:18.498223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.498194    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" event={"ID":"bad9cec6-091a-48a4-afbf-2c86542c0df2","Type":"ContainerStarted","Data":"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"}
Oct 30 20:16:18.498333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.498241    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" event={"ID":"bad9cec6-091a-48a4-afbf-2c86542c0df2","Type":"ContainerStarted","Data":"8499021afb44710fe9d6e45d91f4930ee40031d9a06d254e3bc432d1fed084d6"}
Oct 30 20:16:18.498333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.498269    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:18.505039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.505020    5085 scope.go:122] "RemoveContainer" containerID="1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"
Oct 30 20:16:18.505349 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:18.505313    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e\": container with ID starting with 1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e not found: ID does not exist" containerID="1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"
Oct 30 20:16:18.505470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.505353    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e"} err="failed to get container status \"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e\": rpc error: code = NotFound desc = could not find container \"1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e\": container with ID starting with 1cd7b6046073ebd702960a2428be2db3830e4ff8dd15cc15e733ae7f9340b30e not found: ID does not exist"
Oct 30 20:16:18.505470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.505375    5085 scope.go:122] "RemoveContainer" containerID="6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"
Oct 30 20:16:18.514616 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.514601    5085 scope.go:122] "RemoveContainer" containerID="6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"
Oct 30 20:16:18.514910 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:18.514889    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c\": container with ID starting with 6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c not found: ID does not exist" containerID="6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"
Oct 30 20:16:18.514961 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.514917    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c"} err="failed to get container status \"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c\": rpc error: code = NotFound desc = could not find container \"6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c\": container with ID starting with 6d9ec3a9d3854e359b6a835d75c825d8670b9ade108a434665ad5869e3a5fc2c not found: ID does not exist"
Oct 30 20:16:18.657133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.657085    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" podStartSLOduration=2.65707081 podStartE2EDuration="2.65707081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:18.65464371 +0000 UTC m=+3088.159047457" watchObservedRunningTime="2026-10-30 20:16:18.65707081 +0000 UTC m=+3088.161474556"
Oct 30 20:16:18.767799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.767695    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" podStartSLOduration=1.767682156 podStartE2EDuration="1.767682156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:18.765587254 +0000 UTC m=+3088.269990991" watchObservedRunningTime="2026-10-30 20:16:18.767682156 +0000 UTC m=+3088.272085903"
Oct 30 20:16:18.873014 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.872981    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:16:18.878878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.878836    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f53eb9-597ccf9699-jz6pj"]
Oct 30 20:16:18.984044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.984009    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:16:18.990628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:18.990595    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f53eb9-5c675c76bb-xks4x"]
Oct 30 20:16:20.633338 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:20.633295    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" path="/var/lib/kubelet/pods/7d5c323e-bfc2-47a8-85fc-ef9a96537aaa/volumes"
Oct 30 20:16:20.633920 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:20.633898    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" path="/var/lib/kubelet/pods/f996f1e3-ae0e-4f90-9e93-86ce7e747db0/volumes"
Oct 30 20:16:21.939161 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:21.939128    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:16:25.046023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.045994    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"]
Oct 30 20:16:25.046594 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046579    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" containerName="payload-processing"
Oct 30 20:16:25.046666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046595    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" containerName="payload-processing"
Oct 30 20:16:25.046666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046627    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" containerName="payload-pre-processing"
Oct 30 20:16:25.046666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046633    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" containerName="payload-pre-processing"
Oct 30 20:16:25.046765 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046729    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5c323e-bfc2-47a8-85fc-ef9a96537aaa" containerName="payload-processing"
Oct 30 20:16:25.046765 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.046742    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f996f1e3-ae0e-4f90-9e93-86ce7e747db0" containerName="payload-pre-processing"
Oct 30 20:16:25.051800 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.051781    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.103258 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.103229    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:16:25.107460 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.107433    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"]
Oct 30 20:16:25.149109 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.149085    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:16:25.182090 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.182065    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.182231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.182141    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjq7n\" (UniqueName: \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.182273 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.182241    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.284029 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.283993    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.284211 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.284183    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.284257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.284247    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjq7n\" (UniqueName: \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.284789 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.284765    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.286681 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.286655    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.346021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.345952    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjq7n\" (UniqueName: \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\") pod \"maas-api-revoke-sub-7120eee0-jr66p\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.365235 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.365206    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:25.740995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:25.740959    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"]
Oct 30 20:16:25.741762 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:25.741731    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13402d8_bdcc_49f4_bf02_0ef0b477fdae.slice/crio-91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f WatchSource:0}: Error finding container 91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f: Status 404 returned error can't find the container with id 91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f
Oct 30 20:16:26.560928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:26.560887    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p" event={"ID":"e13402d8-bdcc-49f4-bf02-0ef0b477fdae","Type":"ContainerStarted","Data":"c95b86ae2c8f166b0fd7002ac0c8731b95f41ac26bcf2ba03e06ba65b49e6ec4"}
Oct 30 20:16:26.561336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:26.560956    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p" event={"ID":"e13402d8-bdcc-49f4-bf02-0ef0b477fdae","Type":"ContainerStarted","Data":"91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f"}
Oct 30 20:16:26.671853 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:26.671803    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p" podStartSLOduration=2.671783744 podStartE2EDuration="2.671783744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:26.671176143 +0000 UTC m=+3096.175579886" watchObservedRunningTime="2026-10-30 20:16:26.671783744 +0000 UTC m=+3096.176187492"
Oct 30 20:16:27.569605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:27.569553    5085 generic.go:356] "Generic (PLEG): container finished" podID="e13402d8-bdcc-49f4-bf02-0ef0b477fdae" containerID="c95b86ae2c8f166b0fd7002ac0c8731b95f41ac26bcf2ba03e06ba65b49e6ec4" exitCode=0
Oct 30 20:16:27.570009 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:27.569608    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p" event={"ID":"e13402d8-bdcc-49f4-bf02-0ef0b477fdae","Type":"ContainerDied","Data":"c95b86ae2c8f166b0fd7002ac0c8731b95f41ac26bcf2ba03e06ba65b49e6ec4"}
Oct 30 20:16:28.313809 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.313771    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:16:28.317992 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.317975    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.367737 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.367705    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-0b9215-serving-cert\""
Oct 30 20:16:28.418478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.418447    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:16:28.419457 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.419432    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75qr\" (UniqueName: \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.419607 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.419483    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.419667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.419634    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.419719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.419690    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.520837 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.520804    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.521021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.520924    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75qr\" (UniqueName: \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.521021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.520989    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.521110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.521054    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.521176 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:28.521158    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-serving-cert: secret "maas-api-e2e-praxis-b-0b9215-serving-cert" not found
Oct 30 20:16:28.521243 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:28.521231    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls podName:659ecac3-50cb-469c-954a-3320295ea4b4 nodeName:}" failed. No retries permitted until 2026-10-30 20:16:29.021211208 +0000 UTC m=+3098.525614948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls") pod "maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" (UID: "659ecac3-50cb-469c-954a-3320295ea4b4") : secret "maas-api-e2e-praxis-b-0b9215-serving-cert" not found
Oct 30 20:16:28.521640 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.521620    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.524003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.523978    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.579767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.579742    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75qr\" (UniqueName: \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:28.734558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.734534    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:28.824800 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.824768    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\") pod \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") "
Oct 30 20:16:28.824969 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.824907    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\") pod \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") "
Oct 30 20:16:28.825010 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.824986    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\" (UniqueName: \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\") pod \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\" (UID: \"e13402d8-bdcc-49f4-bf02-0ef0b477fdae\") "
Oct 30 20:16:28.825311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.825284    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca" pod "e13402d8-bdcc-49f4-bf02-0ef0b477fdae" (UID: "e13402d8-bdcc-49f4-bf02-0ef0b477fdae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:28.825557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.825540    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:28.827009 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.826982    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca" pod "e13402d8-bdcc-49f4-bf02-0ef0b477fdae" (UID: "e13402d8-bdcc-49f4-bf02-0ef0b477fdae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:28.827141 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.827118    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n" pod "e13402d8-bdcc-49f4-bf02-0ef0b477fdae" (UID: "e13402d8-bdcc-49f4-bf02-0ef0b477fdae"). InnerVolumeSpecName "kube-api-access-xjq7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:28.926415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.926379    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:28.926415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:28.926411    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjq7n\" (UniqueName: \"kubernetes.io/projected/e13402d8-bdcc-49f4-bf02-0ef0b477fdae-kube-api-access-xjq7n\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:29.027324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.027284    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:29.029668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.029647    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:29.232272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.232238    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:29.511393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.511305    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:16:29.511548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.511473    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:16:29.587704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.587677    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"
Oct 30 20:16:29.588085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.587721    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p" event={"ID":"e13402d8-bdcc-49f4-bf02-0ef0b477fdae","Type":"ContainerDied","Data":"91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f"}
Oct 30 20:16:29.588085 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.587765    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a4889a486974fde25e13da1a0d8b56509459869d78b12f5f74123be3e8a39f"
Oct 30 20:16:29.610337 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:29.610311    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659ecac3_50cb_469c_954a_3320295ea4b4.slice/crio-36bc6424c1702cae45185b65c785c1660bed2a1921d4194ef4ad801247cf5198 WatchSource:0}: Error finding container 36bc6424c1702cae45185b65c785c1660bed2a1921d4194ef4ad801247cf5198: Status 404 returned error can't find the container with id 36bc6424c1702cae45185b65c785c1660bed2a1921d4194ef4ad801247cf5198
Oct 30 20:16:29.611694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.611671    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:16:29.611912 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:29.611896    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:16:30.597013 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:30.596974    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" event={"ID":"659ecac3-50cb-469c-954a-3320295ea4b4","Type":"ContainerStarted","Data":"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"}
Oct 30 20:16:30.597483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:30.597058    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" event={"ID":"659ecac3-50cb-469c-954a-3320295ea4b4","Type":"ContainerStarted","Data":"36bc6424c1702cae45185b65c785c1660bed2a1921d4194ef4ad801247cf5198"}
Oct 30 20:16:30.597483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:30.597080    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:30.714296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:30.714239    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" podStartSLOduration=1.9191868890000001 podStartE2EDuration="2.714225791s" totalImagesPullingTime="795.038902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:30.710810677 +0000 UTC m=+3100.215214427" watchObservedRunningTime="2026-10-30 20:16:30.714225791 +0000 UTC m=+3100.218629551"
Oct 30 20:16:32.606805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.606770    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:16:32.607251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.607028    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" podUID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" containerName="maas-api" containerID="cri-o://9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48" gracePeriod=30
Oct 30 20:16:32.685524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.685491    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:16:32.869283 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.869225    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:16:32.971113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.971071    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") pod \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") "
Oct 30 20:16:32.971310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.971171    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\") pod \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") "
Oct 30 20:16:32.971310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.971276    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\") pod \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") "
Oct 30 20:16:32.971454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.971345    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\") pod \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\" (UID: \"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2\") "
Oct 30 20:16:32.971816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.971786    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca" pod "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" (UID: "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:32.973682 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.973657    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls" pod "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" (UID: "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:32.974129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.974103    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh" pod "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" (UID: "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2"). InnerVolumeSpecName "kube-api-access-pjflh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:32.974215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:32.974181    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs" pod "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" (UID: "ea7c11c2-93ad-4a3d-8960-0f39aa993bd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:33.001765 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.001723    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:16:33.002669 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002642    5085 cpu_manager.go:414] "containerMap: removing container" podUID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" containerName="maas-api"
Oct 30 20:16:33.002669 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002666    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" containerName="maas-api"
Oct 30 20:16:33.002855 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002700    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e13402d8-bdcc-49f4-bf02-0ef0b477fdae" containerName="revoke-keys"
Oct 30 20:16:33.002855 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002708    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13402d8-bdcc-49f4-bf02-0ef0b477fdae" containerName="revoke-keys"
Oct 30 20:16:33.002946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002880    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13402d8-bdcc-49f4-bf02-0ef0b477fdae" containerName="revoke-keys"
Oct 30 20:16:33.002946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.002907    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" containerName="maas-api"
Oct 30 20:16:33.007001 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.006980    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.073257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.073228    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:33.073257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.073253    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjflh\" (UniqueName: \"kubernetes.io/projected/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-kube-api-access-pjflh\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:33.073257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.073265    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:33.073257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.073274    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:16:33.083394 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.083366    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:16:33.174672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.174629    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.174892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.174748    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175102 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175074    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175122    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8cz\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175149    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175278    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175340    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175411    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.175659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.175461    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276323    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276418    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276477    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276525    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8cz\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276545    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276633    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276727    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276768    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.276811    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.277250    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.277510    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.277720    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.277720    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.279590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.278341    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.280535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.279949    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.280535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.280187    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.334776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.334734    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.335001 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.334978    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8cz\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.661636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.660767    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:33.684576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.684526    5085 generic.go:356] "Generic (PLEG): container finished" podID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" containerID="9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48" exitCode=0
Oct 30 20:16:33.684816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.684791    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" event={"ID":"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2","Type":"ContainerDied","Data":"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"}
Oct 30 20:16:33.684964 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.684947    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v" event={"ID":"ea7c11c2-93ad-4a3d-8960-0f39aa993bd2","Type":"ContainerDied","Data":"d8fd5f8627b95a821cdc641c50d915a6be95d014da3ca83efde5a0cae9e7acc8"}
Oct 30 20:16:33.685098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.685084    5085 scope.go:122] "RemoveContainer" containerID="9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"
Oct 30 20:16:33.685467 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.685439    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"
Oct 30 20:16:33.730807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.730779    5085 scope.go:122] "RemoveContainer" containerID="9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"
Oct 30 20:16:33.733150 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:33.731953    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48\": container with ID starting with 9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48 not found: ID does not exist" containerID="9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"
Oct 30 20:16:33.733150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.732004    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48"} err="failed to get container status \"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48\": rpc error: code = NotFound desc = could not find container \"9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48\": container with ID starting with 9c72b8ae650e9fbc8e040883fa5215271fa8e0f75e93e8e9764055c22369ea48 not found: ID does not exist"
Oct 30 20:16:33.855378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.855334    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:16:33.861209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:33.861171    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f53eb9-598f77d94d-xgj8v"]
Oct 30 20:16:34.156242 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.156210    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:16:34.156393 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:34.156286    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e0d6e4_7905_41e5_8d6d_c185ade9549d.slice/crio-b5a5b6d0bda506d0bfe37042a7ae5016fab7f8688635141c0f2f10591507b409 WatchSource:0}: Error finding container b5a5b6d0bda506d0bfe37042a7ae5016fab7f8688635141c0f2f10591507b409: Status 404 returned error can't find the container with id b5a5b6d0bda506d0bfe37042a7ae5016fab7f8688635141c0f2f10591507b409
Oct 30 20:16:34.159475 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.159434    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:34.159609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.159518    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:34.159680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.159633    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:34.632052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.632008    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7c11c2-93ad-4a3d-8960-0f39aa993bd2" path="/var/lib/kubelet/pods/ea7c11c2-93ad-4a3d-8960-0f39aa993bd2/volumes"
Oct 30 20:16:34.701838 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.701796    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" event={"ID":"06e0d6e4-7905-41e5-8d6d-c185ade9549d","Type":"ContainerStarted","Data":"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"}
Oct 30 20:16:34.702359 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.701871    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" event={"ID":"06e0d6e4-7905-41e5-8d6d-c185ade9549d","Type":"ContainerStarted","Data":"b5a5b6d0bda506d0bfe37042a7ae5016fab7f8688635141c0f2f10591507b409"}
Oct 30 20:16:34.820995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:34.820943    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podStartSLOduration=2.82093007 podStartE2EDuration="2.82093007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:34.816380503 +0000 UTC m=+3104.320784287" watchObservedRunningTime="2026-10-30 20:16:34.82093007 +0000 UTC m=+3104.325333817"
Oct 30 20:16:35.681590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:35.680642    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:35.681850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:35.681817    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:35.681967 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:35.681881    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:36.610763 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:36.610731    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:16:36.678588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:36.677950    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:36.678588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:36.678009    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:37.679594 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:37.678886    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:37.679594 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:37.678958    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:38.682033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:38.681990    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:38.682524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:38.682045    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:39.680587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:39.676948    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:39.680587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:39.677013    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:40.661937 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:40.661897    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Oct 30 20:16:40.662334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:40.661959    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Oct 30 20:16:41.666311 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:41.666283    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:41.666787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:41.666769    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:41.667298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:41.667283    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:16:43.871317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.871283    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:16:43.876826 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.876799    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.946557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.946524    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:16:43.991930 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.991890    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.992130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992076    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.992220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992162    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.992220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992198    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.993898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992235    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.993898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992254    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.993898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.992273    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.993898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.993806    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:43.994165 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:43.994041    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7spk\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.095776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095743    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.095776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095781    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095800    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095863    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095904    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7spk\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.095937    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096019    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096055    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096088    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096239    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096492    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096692    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096693    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.096828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.096772    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.098285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.098264    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.098431 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.098414    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.156979 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.156900    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.157148 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.157103    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7spk\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\") pod \"e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") " pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.204636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.204601    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:44.637257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.637222    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:16:44.637421 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:44.637393    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba37489_7b14_4b66_b0c5_9778066b65c6.slice/crio-713523ace9d5260f75b473bd3a3c3862364a18821fc709f99cb9cb459c969474 WatchSource:0}: Error finding container 713523ace9d5260f75b473bd3a3c3862364a18821fc709f99cb9cb459c969474: Status 404 returned error can't find the container with id 713523ace9d5260f75b473bd3a3c3862364a18821fc709f99cb9cb459c969474
Oct 30 20:16:44.639764 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.639729    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:44.639847 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.639799    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:44.639847 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.639835    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:44.745148 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.745112    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:16:44.749664 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.749643    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.787571 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.787519    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" event={"ID":"eba37489-7b14-4b66-b0c5-9778066b65c6","Type":"ContainerStarted","Data":"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"}
Oct 30 20:16:44.787756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.787590    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" event={"ID":"eba37489-7b14-4b66-b0c5-9778066b65c6","Type":"ContainerStarted","Data":"713523ace9d5260f75b473bd3a3c3862364a18821fc709f99cb9cb459c969474"}
Oct 30 20:16:44.800865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.800796    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-487c5f\""
Oct 30 20:16:44.800865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.800844    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-487c5f-dockercfg-zdp5w\""
Oct 30 20:16:44.802919 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.802898    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:16:44.804504 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.804479    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5hg\" (UniqueName: \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.804678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.804656    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.887300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.887207    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:16:44.897470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.896723    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:44.905657 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.905615    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5hg\" (UniqueName: \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.905828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.905788    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.908398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.908370    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.935812 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.935782    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:16:44.973244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.973211    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5hg\" (UniqueName: \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\") pod \"payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:44.993218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:44.993165    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podStartSLOduration=1.9931497550000001 podStartE2EDuration="1.993149755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:44.989079767 +0000 UTC m=+3114.493483513" watchObservedRunningTime="2026-10-30 20:16:44.993149755 +0000 UTC m=+3114.497553501"
Oct 30 20:16:45.006675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.006637    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvmz9\" (UniqueName: \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.006877 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.006785    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.065104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.065066    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:45.112124 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.112051    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvmz9\" (UniqueName: \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.112310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.112219    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.113353 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.113325    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.173878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.173816    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvmz9\" (UniqueName: \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\") pod \"payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.207484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.207448    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:45.211649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.211605    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:45.211883 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.211674    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:45.216855 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.214730    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.517240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.517211    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:16:45.517828 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:45.517764    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a79f69_2a85_4fea_8037_aa339780c68d.slice/crio-8e5c4e911ea54a5626717f324172bd1427ea06ede56e181825f90bba0f7d3737 WatchSource:0}: Error finding container 8e5c4e911ea54a5626717f324172bd1427ea06ede56e181825f90bba0f7d3737: Status 404 returned error can't find the container with id 8e5c4e911ea54a5626717f324172bd1427ea06ede56e181825f90bba0f7d3737
Oct 30 20:16:45.620065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.620039    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:16:45.620440 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:45.620411    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989f3472_ef03_4865_8d85_7e5388757631.slice/crio-2e71e30e7b7b8194008827f0571d2f00294c171ce357c1bca218b80c859b6c10 WatchSource:0}: Error finding container 2e71e30e7b7b8194008827f0571d2f00294c171ce357c1bca218b80c859b6c10: Status 404 returned error can't find the container with id 2e71e30e7b7b8194008827f0571d2f00294c171ce357c1bca218b80c859b6c10
Oct 30 20:16:45.796480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.796393    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" event={"ID":"00a79f69-2a85-4fea-8037-aa339780c68d","Type":"ContainerStarted","Data":"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"}
Oct 30 20:16:45.796480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.796457    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" event={"ID":"00a79f69-2a85-4fea-8037-aa339780c68d","Type":"ContainerStarted","Data":"8e5c4e911ea54a5626717f324172bd1427ea06ede56e181825f90bba0f7d3737"}
Oct 30 20:16:45.796480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.796485    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:45.798309 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.798274    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" event={"ID":"989f3472-ef03-4865-8d85-7e5388757631","Type":"ContainerStarted","Data":"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"}
Oct 30 20:16:45.798443 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.798317    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" event={"ID":"989f3472-ef03-4865-8d85-7e5388757631","Type":"ContainerStarted","Data":"2e71e30e7b7b8194008827f0571d2f00294c171ce357c1bca218b80c859b6c10"}
Oct 30 20:16:45.798494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.798480    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:45.910386 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:45.910335    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" podStartSLOduration=1.910322749 podStartE2EDuration="1.910322749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:45.906912379 +0000 UTC m=+3115.411316123" watchObservedRunningTime="2026-10-30 20:16:45.910322749 +0000 UTC m=+3115.414726495"
Oct 30 20:16:46.041477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:46.041421    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" podStartSLOduration=2.04140474 podStartE2EDuration="2.04140474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:46.037425174 +0000 UTC m=+3115.541828923" watchObservedRunningTime="2026-10-30 20:16:46.04140474 +0000 UTC m=+3115.545808486"
Oct 30 20:16:46.204945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:46.204915    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:46.205135 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:46.204977    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:47.205393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:47.205356    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:47.205799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:47.205417    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:48.122134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.122098    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:16:48.126201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.126180    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.205455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.205421    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:48.206019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.205499    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:48.227045 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.227017    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:16:48.249296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249255    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249323    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvxj\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249452    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249494    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249681    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249721    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249753    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249852    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.249928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.249890    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350751 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350715    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350765    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350784    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350846    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350912    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvxj\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.350970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350968    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.350998    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351141    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351182    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351216    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351287    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351422    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351551 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351531    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.351908 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.351891    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.353273 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.353253    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.353418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.353400    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.413379 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.413297    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvxj\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.424596 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.424558    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\") pod \"e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") " pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.440649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.440611    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:48.918520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.918487    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:16:48.919478 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:48.919449    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82777244_3c44_4370_bb57_3126b5777e2e.slice/crio-b4253dabafba7aaed7543ee2007876bd3dd4bd3e3dcbe2637f204f74d82b975c WatchSource:0}: Error finding container b4253dabafba7aaed7543ee2007876bd3dd4bd3e3dcbe2637f204f74d82b975c: Status 404 returned error can't find the container with id b4253dabafba7aaed7543ee2007876bd3dd4bd3e3dcbe2637f204f74d82b975c
Oct 30 20:16:48.921972 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.921941    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:48.922051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.922002    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:48.922051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:48.922033    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:16:49.205416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.205375    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:49.205662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.205442    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:49.830706 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.830673    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" event={"ID":"82777244-3c44-4370-bb57-3126b5777e2e","Type":"ContainerStarted","Data":"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"}
Oct 30 20:16:49.830896 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.830725    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" event={"ID":"82777244-3c44-4370-bb57-3126b5777e2e","Type":"ContainerStarted","Data":"b4253dabafba7aaed7543ee2007876bd3dd4bd3e3dcbe2637f204f74d82b975c"}
Oct 30 20:16:49.891116 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.890677    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:16:49.899222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.899196    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:49.975590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:49.956279    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-cafbb2-dockercfg-g8d9n\""
Oct 30 20:16:50.025589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.023515    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8lz\" (UniqueName: \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.059588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.039861    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-cafbb2\""
Oct 30 20:16:50.059588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.049405    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:16:50.059588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.053685    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.121618 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.121518    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:16:50.127584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.127323    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podStartSLOduration=2.127308531 podStartE2EDuration="2.127308531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.988001403 +0000 UTC m=+3119.492405187" watchObservedRunningTime="2026-10-30 20:16:50.127308531 +0000 UTC m=+3119.631712277"
Oct 30 20:16:50.128136 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.127993    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.132977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.132953    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:16:50.154844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.154817    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.154984 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.154970    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8lz\" (UniqueName: \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.155657 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.155636    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.208780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.208720    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.209316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.208811    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:50.217596 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.216924    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8lz\" (UniqueName: \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\") pod \"payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.249663 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.249634    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:50.256201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.256159    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.268950 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.268907    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmff\" (UniqueName: \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.377643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.372385    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmff\" (UniqueName: \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.377643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.372523    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.377643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.373488    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.443325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.441134    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:50.443325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.442679    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:50.443325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.442742    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:50.443325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.443300    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmff\" (UniqueName: \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\") pod \"payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.447219 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.447194    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:50.746989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.746954    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:16:50.748728 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:50.748698    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad19f68_49ab_42db_82f6_6cc8b2111f47.slice/crio-ea53b19ecbf069f66ed299980b50f6281c816d20d4f51676f98b3fcd052d9299 WatchSource:0}: Error finding container ea53b19ecbf069f66ed299980b50f6281c816d20d4f51676f98b3fcd052d9299: Status 404 returned error can't find the container with id ea53b19ecbf069f66ed299980b50f6281c816d20d4f51676f98b3fcd052d9299
Oct 30 20:16:50.839090 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.839056    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" event={"ID":"dad19f68-49ab-42db-82f6-6cc8b2111f47","Type":"ContainerStarted","Data":"ea53b19ecbf069f66ed299980b50f6281c816d20d4f51676f98b3fcd052d9299"}
Oct 30 20:16:50.944435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:50.944404    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:16:50.946326 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:50.946302    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be81ba1_2653_4ca0_aece_ea26849bd079.slice/crio-674ea0106a62ee907ce7ef2a73a87e0245706ab35772dc0d16fc109c2a699b81 WatchSource:0}: Error finding container 674ea0106a62ee907ce7ef2a73a87e0245706ab35772dc0d16fc109c2a699b81: Status 404 returned error can't find the container with id 674ea0106a62ee907ce7ef2a73a87e0245706ab35772dc0d16fc109c2a699b81
Oct 30 20:16:51.205828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.205732    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.205828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.205788    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused"
Oct 30 20:16:51.441921 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.441883    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:51.442411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.441934    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:51.848541 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.848483    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" event={"ID":"dad19f68-49ab-42db-82f6-6cc8b2111f47","Type":"ContainerStarted","Data":"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"}
Oct 30 20:16:51.848756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.848592    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:16:51.850632 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.850599    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" event={"ID":"7be81ba1-2653-4ca0-aece-ea26849bd079","Type":"ContainerStarted","Data":"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"}
Oct 30 20:16:51.850803 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.850672    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" event={"ID":"7be81ba1-2653-4ca0-aece-ea26849bd079","Type":"ContainerStarted","Data":"674ea0106a62ee907ce7ef2a73a87e0245706ab35772dc0d16fc109c2a699b81"}
Oct 30 20:16:51.850803 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.850726    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:16:51.941630 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:51.941587    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:16:51.974590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:51.972006    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" podStartSLOduration=2.971985931 podStartE2EDuration="2.971985931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:51.969849927 +0000 UTC m=+3121.474253675" watchObservedRunningTime="2026-10-30 20:16:51.971985931 +0000 UTC m=+3121.476389678"
Oct 30 20:16:52.101398 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.101272    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" podStartSLOduration=3.101251408 podStartE2EDuration="3.101251408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:52.099046898 +0000 UTC m=+3121.603450646" watchObservedRunningTime="2026-10-30 20:16:52.101251408 +0000 UTC m=+3121.605655157"
Oct 30 20:16:52.243770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.243737    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:52.245245 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.245214    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:52.248578 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.246768    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:16:52.443589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.441902    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:52.443589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:52.442028    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:53.441539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:53.441500    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:53.441762 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:53.441590    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:54.441626 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.441513    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:54.441626 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.441589    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:54.733548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.733476    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:16:54.739223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.739203    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.794426 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.794392    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.794600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.794492    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.794600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.794542    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.798748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.798718    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt2x\" (UniqueName: \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.836685 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.836657    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-547082-serving-cert\""
Oct 30 20:16:54.840542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.840520    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:16:54.900204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.900165    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.900494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.900477    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt2x\" (UniqueName: \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.900752 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.900735    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.900966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.900938    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.901107 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:54.901091    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-serving-cert: secret "maas-api-e2e-shared-a-w0-547082-serving-cert" not found
Oct 30 20:16:54.901178 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:16:54.901155    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls podName:326632a3-36af-4fee-9a49-f32ad2114b56 nodeName:}" failed. No retries permitted until 2026-10-30 20:16:55.401133346 +0000 UTC m=+3124.905537077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls") pod "maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" (UID: "326632a3-36af-4fee-9a49-f32ad2114b56") : secret "maas-api-e2e-shared-a-w0-547082-serving-cert" not found
Oct 30 20:16:54.902193 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.902164    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:54.906097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.906072    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:55.000201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:54.999456    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt2x\" (UniqueName: \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:55.407407 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:55.407309    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:55.409807 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:55.409784    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:55.441905 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:55.441875    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused" start-of-body=
Oct 30 20:16:55.442265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:55.441925    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.243:15021/healthz/ready\": dial tcp 10.132.0.243:15021: connect: connection refused"
Oct 30 20:16:55.654139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:55.654095    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:56.044395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.044366    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:16:56.045430 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:16:56.045405    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326632a3_36af_4fee_9a49_f32ad2114b56.slice/crio-684d490cbec1b403a03094340181bbadcb7c0701d2fefa85151c643a616ad667 WatchSource:0}: Error finding container 684d490cbec1b403a03094340181bbadcb7c0701d2fefa85151c643a616ad667: Status 404 returned error can't find the container with id 684d490cbec1b403a03094340181bbadcb7c0701d2fefa85151c643a616ad667
Oct 30 20:16:56.445327 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.445298    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:56.445786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.445768    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:56.446328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.446308    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:16:56.809159 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.809087    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:16:56.809310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.809164    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:16:56.918007 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:56.917970    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" event={"ID":"326632a3-36af-4fee-9a49-f32ad2114b56","Type":"ContainerStarted","Data":"684d490cbec1b403a03094340181bbadcb7c0701d2fefa85151c643a616ad667"}
Oct 30 20:16:57.935910 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:57.935865    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" event={"ID":"326632a3-36af-4fee-9a49-f32ad2114b56","Type":"ContainerStarted","Data":"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"}
Oct 30 20:16:57.936610 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:57.936549    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:16:58.052019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:16:58.051964    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" podStartSLOduration=2.622646484 podStartE2EDuration="4.051946388s" totalImagesPullingTime="1.429299904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:58.047785531 +0000 UTC m=+3127.552189282" watchObservedRunningTime="2026-10-30 20:16:58.051946388 +0000 UTC m=+3127.556350136"
Oct 30 20:17:02.887436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:02.887402    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:17:02.888232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:02.888212    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:17:04.966600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:04.966557    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:17:12.892220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:12.892134    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:17:21.940156 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:21.940102    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:17:23.271270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.271235    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:17:23.276745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.276722    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.374024 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.373991    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-487c5f-serving-cert\""
Oct 30 20:17:23.375911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.375890    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.376025 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.375955    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.376122 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.376107    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqcl\" (UniqueName: \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.376166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.376146    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.378231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.378208    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.477202    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqcl\" (UniqueName: \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.477267    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.477331    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.477393    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:23.477551    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-serving-cert: secret "maas-api-e2e-shared-a-w3-487c5f-serving-cert" not found
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:23.477636    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls podName:0661eb63-fc1b-4a03-8c02-a80320eb725d nodeName:}" failed. No retries permitted until 2026-10-30 20:17:23.977615699 +0000 UTC m=+3153.482019427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls") pod "maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" (UID: "0661eb63-fc1b-4a03-8c02-a80320eb725d") : secret "maas-api-e2e-shared-a-w3-487c5f-serving-cert" not found
Oct 30 20:17:23.483461 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.478495    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.484884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.484829    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.537949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.537874    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqcl\" (UniqueName: \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.984492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.984457    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:23.987033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:23.987001    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:24.190687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:24.190646    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:24.598853 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:24.598815    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:17:24.600084 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:17:24.600056    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0661eb63_fc1b_4a03_8c02_a80320eb725d.slice/crio-07b3e10fefde29b6bcb5479733fdb01ef66bf1ad7f56b96e77922c5781cfe33d WatchSource:0}: Error finding container 07b3e10fefde29b6bcb5479733fdb01ef66bf1ad7f56b96e77922c5781cfe33d: Status 404 returned error can't find the container with id 07b3e10fefde29b6bcb5479733fdb01ef66bf1ad7f56b96e77922c5781cfe33d
Oct 30 20:17:25.145623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:25.145590    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" event={"ID":"0661eb63-fc1b-4a03-8c02-a80320eb725d","Type":"ContainerStarted","Data":"07b3e10fefde29b6bcb5479733fdb01ef66bf1ad7f56b96e77922c5781cfe33d"}
Oct 30 20:17:26.155341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:26.155305    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" event={"ID":"0661eb63-fc1b-4a03-8c02-a80320eb725d","Type":"ContainerStarted","Data":"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"}
Oct 30 20:17:26.155878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:26.155390    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:26.268709 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:26.268650    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" podStartSLOduration=2.444017423 podStartE2EDuration="3.268634783s" totalImagesPullingTime="824.61736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:26.267348279 +0000 UTC m=+3155.771752047" watchObservedRunningTime="2026-10-30 20:17:26.268634783 +0000 UTC m=+3155.773038530"
Oct 30 20:17:27.164106 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.164076    5085 generic.go:356] "Generic (PLEG): container finished" podID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerID="784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1" exitCode=0
Oct 30 20:17:27.164638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.164151    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerDied","Data":"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"}
Oct 30 20:17:27.164701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.164648    5085 scope.go:122] "RemoveContainer" containerID="784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"
Oct 30 20:17:27.166034 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.166007    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:17:27.166104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.166073    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:17:27.166144 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:27.166102    5085 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"}
Oct 30 20:17:28.192757 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:28.192718    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerStarted","Data":"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"}
Oct 30 20:17:28.497873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:28.497786    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:17:28.498844 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:28.498817    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:28.498966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:28.498887    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:29.498470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:29.498438    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:29.498909 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:29.498497    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:30.497948 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:30.497911    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:30.498139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:30.497993    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:31.203271 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:31.203234    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:17:31.497914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:31.497832    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:31.497914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:31.497890    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:32.183196 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:32.183167    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:17:32.498827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:32.498741    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:32.498827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:32.498798    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:33.311834 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.311801    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:17:33.315794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.315774    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.338960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.338929    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.338960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.338961    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.339129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.339071    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pl7\" (UniqueName: \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.339230 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.339209    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.384586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.383856    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-cafbb2-serving-cert\""
Oct 30 20:17:33.390115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.390090    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:17:33.440749 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.440718    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.440953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.440795    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.440953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.440815    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.440953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.440871    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pl7\" (UniqueName: \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.441545 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:33.441262    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-serving-cert: secret "maas-api-e2e-shared-a-w1-cafbb2-serving-cert" not found
Oct 30 20:17:33.441545 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:33.441340    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls podName:e5389f98-ee52-4bee-b393-cdb7ad0836f2 nodeName:}" failed. No retries permitted until 2026-10-30 20:17:33.941317152 +0000 UTC m=+3163.445720902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls") pod "maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" (UID: "e5389f98-ee52-4bee-b393-cdb7ad0836f2") : secret "maas-api-e2e-shared-a-w1-cafbb2-serving-cert" not found
Oct 30 20:17:33.441983 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.441958    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.443957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.443932    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.499307 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.499274    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body=
Oct 30 20:17:33.499792 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.499338    5085 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused"
Oct 30 20:17:33.501763 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.501736    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pl7\" (UniqueName: \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.950183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.950144    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:33.952648 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:33.952628    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:34.229711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:34.229621    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:34.503010 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:34.502942    5085 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:17:34.504036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:34.504018    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:17:34.607712 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:34.607686    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:17:34.608457 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:17:34.608415    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5389f98_ee52_4bee_b393_cdb7ad0836f2.slice/crio-5f949d7c7f033a8933d6620914bf162a84dfa1940c33929f1c4c487069b1e397 WatchSource:0}: Error finding container 5f949d7c7f033a8933d6620914bf162a84dfa1940c33929f1c4c487069b1e397: Status 404 returned error can't find the container with id 5f949d7c7f033a8933d6620914bf162a84dfa1940c33929f1c4c487069b1e397
Oct 30 20:17:35.248467 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:35.248428    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" event={"ID":"e5389f98-ee52-4bee-b393-cdb7ad0836f2","Type":"ContainerStarted","Data":"5f949d7c7f033a8933d6620914bf162a84dfa1940c33929f1c4c487069b1e397"}
Oct 30 20:17:36.257264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:36.257227    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" event={"ID":"e5389f98-ee52-4bee-b393-cdb7ad0836f2","Type":"ContainerStarted","Data":"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"}
Oct 30 20:17:36.257701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:36.257435    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:36.269933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:36.269899    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:17:36.374149 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:36.374098    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" podStartSLOduration=2.439262461 podStartE2EDuration="3.374083656s" totalImagesPullingTime="934.821195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:36.371892676 +0000 UTC m=+3165.876296422" watchObservedRunningTime="2026-10-30 20:17:36.374083656 +0000 UTC m=+3165.878487403"
Oct 30 20:17:36.383917 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:36.383884    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:17:41.271053 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.270983    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" podUID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" containerName="payload-processing" containerID="cri-o://27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28" gracePeriod=25
Oct 30 20:17:41.384676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.384609    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" podUID="714cde8b-c850-44ab-9fce-c2675c3e28b4" containerName="payload-pre-processing" containerID="cri-o://b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c" gracePeriod=25
Oct 30 20:17:41.560979 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.560955    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:17:41.646101 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.646076    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\" (UniqueName: \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\") pod \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") "
Oct 30 20:17:41.646238 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.646130    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\") pod \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\" (UID: \"a3df9cda-6022-41ea-9a7b-9b267dcb7828\") "
Oct 30 20:17:41.646442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.646420    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume" pod "a3df9cda-6022-41ea-9a7b-9b267dcb7828" (UID: "a3df9cda-6022-41ea-9a7b-9b267dcb7828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:41.646640 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.646624    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3df9cda-6022-41ea-9a7b-9b267dcb7828-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.648089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.648069    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm" pod "a3df9cda-6022-41ea-9a7b-9b267dcb7828" (UID: "a3df9cda-6022-41ea-9a7b-9b267dcb7828"). InnerVolumeSpecName "kube-api-access-5jpqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:41.664625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.664598    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:17:41.747663 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.747633    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\") pod \"714cde8b-c850-44ab-9fce-c2675c3e28b4\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") "
Oct 30 20:17:41.747832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.747760    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\" (UniqueName: \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\") pod \"714cde8b-c850-44ab-9fce-c2675c3e28b4\" (UID: \"714cde8b-c850-44ab-9fce-c2675c3e28b4\") "
Oct 30 20:17:41.748075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.748054    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume" pod "714cde8b-c850-44ab-9fce-c2675c3e28b4" (UID: "714cde8b-c850-44ab-9fce-c2675c3e28b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:41.748471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.748448    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jpqm\" (UniqueName: \"kubernetes.io/projected/a3df9cda-6022-41ea-9a7b-9b267dcb7828-kube-api-access-5jpqm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.748522 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.748481    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714cde8b-c850-44ab-9fce-c2675c3e28b4-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:41.749774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.749753    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk" pod "714cde8b-c850-44ab-9fce-c2675c3e28b4" (UID: "714cde8b-c850-44ab-9fce-c2675c3e28b4"). InnerVolumeSpecName "kube-api-access-v5rpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:41.849862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:41.849782    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5rpk\" (UniqueName: \"kubernetes.io/projected/714cde8b-c850-44ab-9fce-c2675c3e28b4-kube-api-access-v5rpk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:42.269902 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.269870    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:17:42.303533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.303495    5085 generic.go:356] "Generic (PLEG): container finished" podID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" containerID="27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28" exitCode=0
Oct 30 20:17:42.304291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.303591    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"
Oct 30 20:17:42.304291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.303595    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" event={"ID":"a3df9cda-6022-41ea-9a7b-9b267dcb7828","Type":"ContainerDied","Data":"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"}
Oct 30 20:17:42.304291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.303653    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj" event={"ID":"a3df9cda-6022-41ea-9a7b-9b267dcb7828","Type":"ContainerDied","Data":"a8b68414951c0659071ac3c774b6d337717cbdd7dcc7d9b7188652a5587e4b95"}
Oct 30 20:17:42.304291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.303686    5085 scope.go:122] "RemoveContainer" containerID="27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"
Oct 30 20:17:42.306082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.306050    5085 generic.go:356] "Generic (PLEG): container finished" podID="714cde8b-c850-44ab-9fce-c2675c3e28b4" containerID="b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c" exitCode=0
Oct 30 20:17:42.306180 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.306091    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"
Oct 30 20:17:42.306257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.306241    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" event={"ID":"714cde8b-c850-44ab-9fce-c2675c3e28b4","Type":"ContainerDied","Data":"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"}
Oct 30 20:17:42.306302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.306283    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp" event={"ID":"714cde8b-c850-44ab-9fce-c2675c3e28b4","Type":"ContainerDied","Data":"a2b1246f54d0c54a1d4330bfb561a1de0b7ed05194b3c6bd0e66a8c0e185a9cd"}
Oct 30 20:17:42.316194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.316174    5085 scope.go:122] "RemoveContainer" containerID="27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"
Oct 30 20:17:42.316491 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:42.316471    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28\": container with ID starting with 27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28 not found: ID does not exist" containerID="27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"
Oct 30 20:17:42.316591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.316498    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28"} err="failed to get container status \"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28\": rpc error: code = NotFound desc = could not find container \"27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28\": container with ID starting with 27a3f06d3843b22438ad8586b6b889692a41707c812cf90e1f51d695ceee0a28 not found: ID does not exist"
Oct 30 20:17:42.316591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.316516    5085 scope.go:122] "RemoveContainer" containerID="b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"
Oct 30 20:17:42.326842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.326824    5085 scope.go:122] "RemoveContainer" containerID="b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"
Oct 30 20:17:42.327133 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:42.327099    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c\": container with ID starting with b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c not found: ID does not exist" containerID="b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"
Oct 30 20:17:42.327225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.327136    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c"} err="failed to get container status \"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c\": rpc error: code = NotFound desc = could not find container \"b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c\": container with ID starting with b23816888f067cdc24b37d0db9439970cf280bce8b3181500141732363ea4d7c not found: ID does not exist"
Oct 30 20:17:42.492675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.492645    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:17:42.501198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.501165    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d5d3ba-796bc7c8bb-mv6pj"]
Oct 30 20:17:42.504831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.504808    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:17:42.505484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505468    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" containerName="payload-processing"
Oct 30 20:17:42.505540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505486    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" containerName="payload-processing"
Oct 30 20:17:42.505540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505513    5085 cpu_manager.go:414] "containerMap: removing container" podUID="714cde8b-c850-44ab-9fce-c2675c3e28b4" containerName="payload-pre-processing"
Oct 30 20:17:42.505540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505519    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714cde8b-c850-44ab-9fce-c2675c3e28b4" containerName="payload-pre-processing"
Oct 30 20:17:42.505680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505636    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" containerName="payload-processing"
Oct 30 20:17:42.505680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.505651    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="714cde8b-c850-44ab-9fce-c2675c3e28b4" containerName="payload-pre-processing"
Oct 30 20:17:42.509038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.509022    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.556333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.556272    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4bm\" (UniqueName: \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.556333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.556312    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.556500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.556346    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.556558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.556536    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.558796 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.558781    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-056a27-serving-cert\""
Oct 30 20:17:42.599961 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.599931    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:17:42.603752 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.603726    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:17:42.608288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.608260    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d5d3ba-6c5fc4fd59-z2fbp"]
Oct 30 20:17:42.635221 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.635185    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714cde8b-c850-44ab-9fce-c2675c3e28b4" path="/var/lib/kubelet/pods/714cde8b-c850-44ab-9fce-c2675c3e28b4/volumes"
Oct 30 20:17:42.635701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.635684    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3df9cda-6022-41ea-9a7b-9b267dcb7828" path="/var/lib/kubelet/pods/a3df9cda-6022-41ea-9a7b-9b267dcb7828/volumes"
Oct 30 20:17:42.658270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.658240    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4bm\" (UniqueName: \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.658430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.658285    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.658430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.658329    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.658588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.658500    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.658662 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:42.658647    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-serving-cert: secret "maas-api-e2e-shared-a-w2-056a27-serving-cert" not found
Oct 30 20:17:42.658728 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:42.658714    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls podName:2fac75e5-dcc7-4feb-b6be-3e37a13a6e41 nodeName:}" failed. No retries permitted until 2026-10-30 20:17:43.158692918 +0000 UTC m=+3172.663096663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls") pod "maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" (UID: "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41") : secret "maas-api-e2e-shared-a-w2-056a27-serving-cert" not found
Oct 30 20:17:42.659420 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.659398    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.660996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.660975    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:42.717075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:42.717043    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4bm\" (UniqueName: \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:43.165048 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:43.165016    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:43.167422 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:43.167403    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:43.421716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:43.421683    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:43.799202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:43.799174    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:17:43.800228 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:17:43.800192    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fac75e5_dcc7_4feb_b6be_3e37a13a6e41.slice/crio-ce2a239d56260cacb9eac02839f1cb15951d59a8c78cb81296fdbbf2c0a13ffa WatchSource:0}: Error finding container ce2a239d56260cacb9eac02839f1cb15951d59a8c78cb81296fdbbf2c0a13ffa: Status 404 returned error can't find the container with id ce2a239d56260cacb9eac02839f1cb15951d59a8c78cb81296fdbbf2c0a13ffa
Oct 30 20:17:44.324436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:44.324398    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" event={"ID":"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41","Type":"ContainerStarted","Data":"ce2a239d56260cacb9eac02839f1cb15951d59a8c78cb81296fdbbf2c0a13ffa"}
Oct 30 20:17:45.335343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:45.335302    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" event={"ID":"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41","Type":"ContainerStarted","Data":"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"}
Oct 30 20:17:45.335851 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:45.335518    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:45.470292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:45.470241    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" podStartSLOduration=2.630978311 podStartE2EDuration="3.47022833s" totalImagesPullingTime="839.250019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:45.466156399 +0000 UTC m=+3174.970560157" watchObservedRunningTime="2026-10-30 20:17:45.47022833 +0000 UTC m=+3174.974632077"
Oct 30 20:17:47.102603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:47.102556    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:17:51.347906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:51.347876    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:17:51.939464 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:51.939427    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:17:56.054873 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.054828    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:17:56.055933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.055098    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" podUID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" containerName="maas-api" containerID="cri-o://362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe" gracePeriod=30
Oct 30 20:17:56.346061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.346031    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:17:56.417871 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.417840    5085 generic.go:356] "Generic (PLEG): container finished" podID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" containerID="362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe" exitCode=0
Oct 30 20:17:56.418047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.417896    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"
Oct 30 20:17:56.418047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.417918    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" event={"ID":"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a","Type":"ContainerDied","Data":"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"}
Oct 30 20:17:56.418047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.417974    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p" event={"ID":"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a","Type":"ContainerDied","Data":"df3949f12aa7835b365db7d44bbe883a63a67836fe319b966d2ea17bb52e9e52"}
Oct 30 20:17:56.418047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.418001    5085 scope.go:122] "RemoveContainer" containerID="362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"
Oct 30 20:17:56.428214 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.428195    5085 scope.go:122] "RemoveContainer" containerID="362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"
Oct 30 20:17:56.428485 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:17:56.428462    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe\": container with ID starting with 362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe not found: ID does not exist" containerID="362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"
Oct 30 20:17:56.428531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.428498    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe"} err="failed to get container status \"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe\": rpc error: code = NotFound desc = could not find container \"362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe\": container with ID starting with 362b5db174b5c768b8e7d1a162aa9a16583d900e24934e02a9d893477b94cdfe not found: ID does not exist"
Oct 30 20:17:56.446173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446142    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\") pod \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") "
Oct 30 20:17:56.446304 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446210    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\") pod \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") "
Oct 30 20:17:56.446381 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446326    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\" (UniqueName: \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\") pod \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") "
Oct 30 20:17:56.446381 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446361    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\") pod \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\" (UID: \"846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a\") "
Oct 30 20:17:56.446575 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446536    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca" pod "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" (UID: "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:56.446881 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.446850    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.448415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.448392    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs" pod "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" (UID: "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:56.448573 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.448536    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls" pod "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" (UID: "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:56.448846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.448827    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g" pod "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" (UID: "846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a"). InnerVolumeSpecName "kube-api-access-7hc7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:56.548200 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.548166    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.548200 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.548202    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc7g\" (UniqueName: \"kubernetes.io/projected/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-kube-api-access-7hc7g\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.548417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.548220    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:17:56.833507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.833472    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:17:56.837330 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:56.837302    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d5d3ba-7846b79948-msg8p"]
Oct 30 20:17:58.631966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:17:58.631885    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" path="/var/lib/kubelet/pods/846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a/volumes"
Oct 30 20:18:19.043332 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:19.043296    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:18:21.939949 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:18:21.939917    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:18:25.606102 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:25.606061    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:18:25.721396 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:25.721338    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:18:30.606419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.606359    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" podUID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" containerName="payload-processing" containerID="cri-o://2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f" gracePeriod=25
Oct 30 20:18:30.722701 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.722665    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" podUID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" containerName="payload-pre-processing" containerID="cri-o://8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d" gracePeriod=25
Oct 30 20:18:30.885455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.885429    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:18:30.994694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.994666    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:18:30.996513 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.996481    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\") pod \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") "
Oct 30 20:18:30.996753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.996733    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\" (UniqueName: \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\") pod \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\" (UID: \"299e2a7c-3c07-4d23-8a78-78ab2a25b996\") "
Oct 30 20:18:30.997215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.996907    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume" pod "299e2a7c-3c07-4d23-8a78-78ab2a25b996" (UID: "299e2a7c-3c07-4d23-8a78-78ab2a25b996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:30.997341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.997322    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/299e2a7c-3c07-4d23-8a78-78ab2a25b996-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:30.999503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:30.999475    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7" pod "299e2a7c-3c07-4d23-8a78-78ab2a25b996" (UID: "299e2a7c-3c07-4d23-8a78-78ab2a25b996"). InnerVolumeSpecName "kube-api-access-6t7q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:31.098719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.098689    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\" (UniqueName: \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\") pod \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") "
Oct 30 20:18:31.098893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.098737    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\") pod \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\" (UID: \"47e53c8f-de9a-4fd4-8181-f8f0b8582a76\") "
Oct 30 20:18:31.099146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.099126    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume" pod "47e53c8f-de9a-4fd4-8181-f8f0b8582a76" (UID: "47e53c8f-de9a-4fd4-8181-f8f0b8582a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:18:31.099249 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.099233    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7q7\" (UniqueName: \"kubernetes.io/projected/299e2a7c-3c07-4d23-8a78-78ab2a25b996-kube-api-access-6t7q7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:31.099288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.099253    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:31.100833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.100816    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c" pod "47e53c8f-de9a-4fd4-8181-f8f0b8582a76" (UID: "47e53c8f-de9a-4fd4-8181-f8f0b8582a76"). InnerVolumeSpecName "kube-api-access-ftg9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:18:31.200669 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.200624    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftg9c\" (UniqueName: \"kubernetes.io/projected/47e53c8f-de9a-4fd4-8181-f8f0b8582a76-kube-api-access-ftg9c\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:18:31.726999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.726966    5085 generic.go:356] "Generic (PLEG): container finished" podID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" containerID="8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d" exitCode=0
Oct 30 20:18:31.727487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.727009    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"
Oct 30 20:18:31.727487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.727049    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" event={"ID":"47e53c8f-de9a-4fd4-8181-f8f0b8582a76","Type":"ContainerDied","Data":"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"}
Oct 30 20:18:31.727487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.727117    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5" event={"ID":"47e53c8f-de9a-4fd4-8181-f8f0b8582a76","Type":"ContainerDied","Data":"5c1dae54d8861ef9211a9b1edbd514ca3022e24869fa0682fc8265e47beb1735"}
Oct 30 20:18:31.727487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.727150    5085 scope.go:122] "RemoveContainer" containerID="8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"
Oct 30 20:18:31.733831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.733806    5085 generic.go:356] "Generic (PLEG): container finished" podID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" containerID="2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f" exitCode=0
Oct 30 20:18:31.733944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.733849    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"
Oct 30 20:18:31.733944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.733884    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" event={"ID":"299e2a7c-3c07-4d23-8a78-78ab2a25b996","Type":"ContainerDied","Data":"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"}
Oct 30 20:18:31.733944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.733929    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh" event={"ID":"299e2a7c-3c07-4d23-8a78-78ab2a25b996","Type":"ContainerDied","Data":"b0c47008ec33708494ac51fbdddb19dd346f590cd415845a33e0fb5c0253b9ad"}
Oct 30 20:18:31.740105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.740091    5085 scope.go:122] "RemoveContainer" containerID="8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"
Oct 30 20:18:31.740356 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:18:31.740338    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d\": container with ID starting with 8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d not found: ID does not exist" containerID="8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"
Oct 30 20:18:31.740399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.740363    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d"} err="failed to get container status \"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d\": rpc error: code = NotFound desc = could not find container \"8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d\": container with ID starting with 8e931f2232cceded61d1aa9783b9c96311843cae5ef8d9854825684bd4fca96d not found: ID does not exist"
Oct 30 20:18:31.740399 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.740379    5085 scope.go:122] "RemoveContainer" containerID="2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"
Oct 30 20:18:31.749724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.749702    5085 scope.go:122] "RemoveContainer" containerID="2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"
Oct 30 20:18:31.749987 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:18:31.749968    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f\": container with ID starting with 2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f not found: ID does not exist" containerID="2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"
Oct 30 20:18:31.750047 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.749993    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f"} err="failed to get container status \"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f\": rpc error: code = NotFound desc = could not find container \"2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f\": container with ID starting with 2b2ef5917c5944386bd78e1580464a823ed3c422d44054cb8608b583e61d429f not found: ID does not exist"
Oct 30 20:18:31.885348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.885266    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:18:31.932097 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:31.932044    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0b9215-6f6588f659-549t5"]
Oct 30 20:18:32.040266 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:32.040230    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:18:32.044989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:32.044960    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0b9215-6f85b674d8-qqnsh"]
Oct 30 20:18:32.633554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:32.633519    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" path="/var/lib/kubelet/pods/299e2a7c-3c07-4d23-8a78-78ab2a25b996/volumes"
Oct 30 20:18:32.633977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:32.633961    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" path="/var/lib/kubelet/pods/47e53c8f-de9a-4fd4-8181-f8f0b8582a76/volumes"
Oct 30 20:18:38.855192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855160    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:18:38.855805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855790    5085 cpu_manager.go:414] "containerMap: removing container" podUID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" containerName="maas-api"
Oct 30 20:18:38.855869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855806    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" containerName="maas-api"
Oct 30 20:18:38.855869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855821    5085 cpu_manager.go:414] "containerMap: removing container" podUID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" containerName="payload-processing"
Oct 30 20:18:38.855869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855826    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" containerName="payload-processing"
Oct 30 20:18:38.855869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855842    5085 cpu_manager.go:414] "containerMap: removing container" podUID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" containerName="payload-pre-processing"
Oct 30 20:18:38.855869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855849    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" containerName="payload-pre-processing"
Oct 30 20:18:38.856073 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855949    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="846bcf37-02fe-4ead-b4c4-2dbc2b08ab1a" containerName="maas-api"
Oct 30 20:18:38.856073 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855960    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="299e2a7c-3c07-4d23-8a78-78ab2a25b996" containerName="payload-processing"
Oct 30 20:18:38.856073 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.855969    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e53c8f-de9a-4fd4-8181-f8f0b8582a76" containerName="payload-pre-processing"
Oct 30 20:18:38.859682 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.859665    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:38.917668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.917633    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-fe49e7-dockercfg-58xsv\""
Oct 30 20:18:38.982746 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.982712    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-fe49e7\""
Oct 30 20:18:38.992970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:38.992924    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:18:39.000838 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.000780    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.001298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.001271    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4vp\" (UniqueName: \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.007762 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.007733    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:18:39.012745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.012689    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.063426 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.063393    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:18:39.102518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.102484    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.102719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.102527    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4vp\" (UniqueName: \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.102719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.102585    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.102863 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.102784    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78npr\" (UniqueName: \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.103279 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.103257    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.161300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.161232    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4vp\" (UniqueName: \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\") pod \"payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.174836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.174813    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.204390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.204355    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78npr\" (UniqueName: \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.204542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.204519    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.205453 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.205416    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.267616 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.267535    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78npr\" (UniqueName: \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\") pod \"payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.332076 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.332044    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.562316 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.562287    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:18:39.562766 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:39.562739    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6f6784_eb00_4cad_b0b3_4365f8446172.slice/crio-ffa89380fc6e90c3cf102e86fc3a4095afe3fb056ba9fde43b01907c8e4b37d9 WatchSource:0}: Error finding container ffa89380fc6e90c3cf102e86fc3a4095afe3fb056ba9fde43b01907c8e4b37d9: Status 404 returned error can't find the container with id ffa89380fc6e90c3cf102e86fc3a4095afe3fb056ba9fde43b01907c8e4b37d9
Oct 30 20:18:39.709895 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:39.709861    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5913b93_1358_42e4_a311_6a1170a00a41.slice/crio-545a23c564c1feca67288110fade14a3bd9356ad9eb021ef339f252186948f42 WatchSource:0}: Error finding container 545a23c564c1feca67288110fade14a3bd9356ad9eb021ef339f252186948f42: Status 404 returned error can't find the container with id 545a23c564c1feca67288110fade14a3bd9356ad9eb021ef339f252186948f42
Oct 30 20:18:39.710653 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.710626    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:18:39.796147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.796116    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" event={"ID":"2e6f6784-eb00-4cad-b0b3-4365f8446172","Type":"ContainerStarted","Data":"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"}
Oct 30 20:18:39.796303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.796174    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" event={"ID":"2e6f6784-eb00-4cad-b0b3-4365f8446172","Type":"ContainerStarted","Data":"ffa89380fc6e90c3cf102e86fc3a4095afe3fb056ba9fde43b01907c8e4b37d9"}
Oct 30 20:18:39.796303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.796204    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:39.798013 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.797984    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" event={"ID":"c5913b93-1358-42e4-a311-6a1170a00a41","Type":"ContainerStarted","Data":"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"}
Oct 30 20:18:39.798139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.798031    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" event={"ID":"c5913b93-1358-42e4-a311-6a1170a00a41","Type":"ContainerStarted","Data":"545a23c564c1feca67288110fade14a3bd9356ad9eb021ef339f252186948f42"}
Oct 30 20:18:39.798139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.798057    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:39.907614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:39.907485    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" podStartSLOduration=1.907470343 podStartE2EDuration="1.907470343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:39.906994387 +0000 UTC m=+3229.411398134" watchObservedRunningTime="2026-10-30 20:18:39.907470343 +0000 UTC m=+3229.411874090"
Oct 30 20:18:40.022662 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:40.022602    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" podStartSLOduration=2.022583545 podStartE2EDuration="2.022583545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:40.018288195 +0000 UTC m=+3229.522691959" watchObservedRunningTime="2026-10-30 20:18:40.022583545 +0000 UTC m=+3229.526987288"
Oct 30 20:18:44.439649 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.439616    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:18:44.444573 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.444542    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.467920 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.467897    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnphq\" (UniqueName: \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.468072 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.468055    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.503485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.503448    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:18:44.567608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.560332    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-b37684-dockercfg-gkzgk\""
Oct 30 20:18:44.567608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.560404    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-b37684\""
Oct 30 20:18:44.567904 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.567813    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:18:44.573066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.571797    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnphq\" (UniqueName: \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.573066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.571972    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.573066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.572848    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.575300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.574396    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.612482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.612449    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:18:44.632395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.632366    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnphq\" (UniqueName: \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\") pod \"payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.673500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.673468    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.673668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.673547    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zs94\" (UniqueName: \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.758919 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.758844    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:44.774284 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.774263    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.774369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.774322    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zs94\" (UniqueName: \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.774984 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.774964    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.833621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.833524    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zs94\" (UniqueName: \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\") pod \"payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:44.893950 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:44.893917    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:45.135265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.134700    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:18:45.268704 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:45.268624    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac207a5_b4f6_4897_b724_5533aa03dac2.slice/crio-0400bdac5c6978b492fd91cab4448018ee72212e7404af917fdd95e59cf524eb WatchSource:0}: Error finding container 0400bdac5c6978b492fd91cab4448018ee72212e7404af917fdd95e59cf524eb: Status 404 returned error can't find the container with id 0400bdac5c6978b492fd91cab4448018ee72212e7404af917fdd95e59cf524eb
Oct 30 20:18:45.268704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.268691    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:18:45.856618 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.856572    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" event={"ID":"5ac207a5-b4f6-4897-b724-5533aa03dac2","Type":"ContainerStarted","Data":"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"}
Oct 30 20:18:45.857056 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.856662    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" event={"ID":"5ac207a5-b4f6-4897-b724-5533aa03dac2","Type":"ContainerStarted","Data":"0400bdac5c6978b492fd91cab4448018ee72212e7404af917fdd95e59cf524eb"}
Oct 30 20:18:45.857056 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.856792    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:45.858353 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.858331    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" event={"ID":"04fda8e3-64e1-4da7-b2dd-9cb95994254b","Type":"ContainerStarted","Data":"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"}
Oct 30 20:18:45.858410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.858372    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" event={"ID":"04fda8e3-64e1-4da7-b2dd-9cb95994254b","Type":"ContainerStarted","Data":"0d5b184baba25155dac95cc8f3f39b61a6138ec0822a0905cab42efb3e7d6e97"}
Oct 30 20:18:45.858410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.858403    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:45.972231 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:45.972176    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" podStartSLOduration=1.972161568 podStartE2EDuration="1.972161568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:45.969220471 +0000 UTC m=+3235.473624218" watchObservedRunningTime="2026-10-30 20:18:45.972161568 +0000 UTC m=+3235.476565315"
Oct 30 20:18:46.083870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:46.083815    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" podStartSLOduration=2.083802695 podStartE2EDuration="2.083802695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:46.08063822 +0000 UTC m=+3235.585041967" watchObservedRunningTime="2026-10-30 20:18:46.083802695 +0000 UTC m=+3235.588206457"
Oct 30 20:18:50.808727 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:50.808700    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:18:50.809106 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:50.808768    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:18:51.939951 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:18:51.939907    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:18:54.973823 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:54.973793    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:18:54.979636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:54.979614    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.056819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.056779    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-64120b\""
Oct 30 20:18:55.060098 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.059975    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:18:55.089714 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.089673    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-64120b-dockercfg-ws7z9\""
Oct 30 20:18:55.121369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.121328    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fskkv\" (UniqueName: \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.121535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.121499    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.189209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.189175    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:18:55.193519 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.193503    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.222529 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.222500    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fskkv\" (UniqueName: \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.222691 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.222635    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.223434 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.223414    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.247754 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.247679    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:18:55.282641 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.282606    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fskkv\" (UniqueName: \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\") pod \"payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.305962 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.305930    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.323939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.323905    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btvr\" (UniqueName: \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.324246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.324225    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.425781 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.425751    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.425975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.425851    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btvr\" (UniqueName: \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.426492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.426469    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.488031 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.487994    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btvr\" (UniqueName: \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\") pod \"payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.507758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.507686    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:55.691333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.691305    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:18:55.691333 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:55.691322    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9738b32_9f30_4648_9f56_a691a8c41a67.slice/crio-374ce953d4a9babb9146b73dae8e61ae2b67382dd66c230cf92e77f3d7c91f94 WatchSource:0}: Error finding container 374ce953d4a9babb9146b73dae8e61ae2b67382dd66c230cf92e77f3d7c91f94: Status 404 returned error can't find the container with id 374ce953d4a9babb9146b73dae8e61ae2b67382dd66c230cf92e77f3d7c91f94
Oct 30 20:18:55.887588 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.887542    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:18:55.888072 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:55.888048    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839c7660_f036_4c85_8f6e_767db2782d14.slice/crio-98fdd2ddc60b2d09adfdcada4b5053e5e8fdad6ec7d823b0d6fc198628f73132 WatchSource:0}: Error finding container 98fdd2ddc60b2d09adfdcada4b5053e5e8fdad6ec7d823b0d6fc198628f73132: Status 404 returned error can't find the container with id 98fdd2ddc60b2d09adfdcada4b5053e5e8fdad6ec7d823b0d6fc198628f73132
Oct 30 20:18:55.933452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.933419    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" event={"ID":"c9738b32-9f30-4648-9f56-a691a8c41a67","Type":"ContainerStarted","Data":"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"}
Oct 30 20:18:55.933736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.933481    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" event={"ID":"c9738b32-9f30-4648-9f56-a691a8c41a67","Type":"ContainerStarted","Data":"374ce953d4a9babb9146b73dae8e61ae2b67382dd66c230cf92e77f3d7c91f94"}
Oct 30 20:18:55.933736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.933506    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:18:55.935043 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:55.935018    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" event={"ID":"839c7660-f036-4c85-8f6e-767db2782d14","Type":"ContainerStarted","Data":"98fdd2ddc60b2d09adfdcada4b5053e5e8fdad6ec7d823b0d6fc198628f73132"}
Oct 30 20:18:56.048108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:56.048011    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" podStartSLOduration=2.04799754 podStartE2EDuration="2.04799754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:56.046940174 +0000 UTC m=+3245.551343923" watchObservedRunningTime="2026-10-30 20:18:56.04799754 +0000 UTC m=+3245.552401286"
Oct 30 20:18:56.868681 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:56.868652    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:18:56.868854 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:56.868718    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:18:56.944246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:56.944212    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" event={"ID":"839c7660-f036-4c85-8f6e-767db2782d14","Type":"ContainerStarted","Data":"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"}
Oct 30 20:18:56.944414 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:56.944301    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:18:57.222660 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:57.222598    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" podStartSLOduration=2.222577784 podStartE2EDuration="2.222577784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:57.21941416 +0000 UTC m=+3246.723817927" watchObservedRunningTime="2026-10-30 20:18:57.222577784 +0000 UTC m=+3246.726981527"
Oct 30 20:18:58.326922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.326881    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:18:58.332347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.332317    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.382705 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.382644    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e4864d-serving-cert\""
Oct 30 20:18:58.387836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.387799    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:18:58.473659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.473617    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.473968 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.473949    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.474143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.474126    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.474257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.474243    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhg5\" (UniqueName: \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.575690 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.575655    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhg5\" (UniqueName: \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.575878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.575827    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.575932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.575886    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.575991 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.575933    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.576620 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.576552    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.578336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.578293    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.578336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.578310    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.634817 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.634785    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhg5\" (UniqueName: \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\") pod \"maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:58.647966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:58.647936    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:18:59.023958 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:59.023897    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:18:59.024519 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:18:59.024493    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec237de_6db4_424b_a4bb_29516c034d7a.slice/crio-0e77dfee208990c866e58ec3e35ae905c87e945bb5716e16078e9d40abcb8b8c WatchSource:0}: Error finding container 0e77dfee208990c866e58ec3e35ae905c87e945bb5716e16078e9d40abcb8b8c: Status 404 returned error can't find the container with id 0e77dfee208990c866e58ec3e35ae905c87e945bb5716e16078e9d40abcb8b8c
Oct 30 20:18:59.969725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:59.969691    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" event={"ID":"7ec237de-6db4-424b-a4bb-29516c034d7a","Type":"ContainerStarted","Data":"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"}
Oct 30 20:18:59.970104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:59.969746    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" event={"ID":"7ec237de-6db4-424b-a4bb-29516c034d7a","Type":"ContainerStarted","Data":"0e77dfee208990c866e58ec3e35ae905c87e945bb5716e16078e9d40abcb8b8c"}
Oct 30 20:18:59.970104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:18:59.969772    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:19:00.137183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:00.137090    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" podStartSLOduration=1.333322934 podStartE2EDuration="2.137075638s" totalImagesPullingTime="803.752704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:00.134087632 +0000 UTC m=+3249.638491378" watchObservedRunningTime="2026-10-30 20:19:00.137075638 +0000 UTC m=+3249.641479384"
Oct 30 20:19:05.984307 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:05.984278    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:19:06.948175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:06.948143    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:19:07.522277 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.522239    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:19:07.522764 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.522485    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" podUID="659ecac3-50cb-469c-954a-3320295ea4b4" containerName="maas-api" containerID="cri-o://acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94" gracePeriod=30
Oct 30 20:19:07.796212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.796180    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:19:07.890687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.890658    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\" (UniqueName: \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\") pod \"659ecac3-50cb-469c-954a-3320295ea4b4\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") "
Oct 30 20:19:07.891018 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.890764    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\") pod \"659ecac3-50cb-469c-954a-3320295ea4b4\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") "
Oct 30 20:19:07.891018 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.890870    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\") pod \"659ecac3-50cb-469c-954a-3320295ea4b4\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") "
Oct 30 20:19:07.891018 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.890902    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") pod \"659ecac3-50cb-469c-954a-3320295ea4b4\" (UID: \"659ecac3-50cb-469c-954a-3320295ea4b4\") "
Oct 30 20:19:07.891379 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.891300    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca" pod "659ecac3-50cb-469c-954a-3320295ea4b4" (UID: "659ecac3-50cb-469c-954a-3320295ea4b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:07.891555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.891530    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/659ecac3-50cb-469c-954a-3320295ea4b4-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:07.892889 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.892860    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs" pod "659ecac3-50cb-469c-954a-3320295ea4b4" (UID: "659ecac3-50cb-469c-954a-3320295ea4b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:07.893039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.893017    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr" pod "659ecac3-50cb-469c-954a-3320295ea4b4" (UID: "659ecac3-50cb-469c-954a-3320295ea4b4"). InnerVolumeSpecName "kube-api-access-c75qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:07.893101 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.893025    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls" pod "659ecac3-50cb-469c-954a-3320295ea4b4" (UID: "659ecac3-50cb-469c-954a-3320295ea4b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:07.954914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.954883    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:19:07.993306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.993273    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:07.993306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.993302    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75qr\" (UniqueName: \"kubernetes.io/projected/659ecac3-50cb-469c-954a-3320295ea4b4-kube-api-access-c75qr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:07.993306 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:07.993314    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/659ecac3-50cb-469c-954a-3320295ea4b4-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:08.029246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.029166    5085 generic.go:356] "Generic (PLEG): container finished" podID="659ecac3-50cb-469c-954a-3320295ea4b4" containerID="acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94" exitCode=0
Oct 30 20:19:08.029246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.029216    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"
Oct 30 20:19:08.029246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.029230    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" event={"ID":"659ecac3-50cb-469c-954a-3320295ea4b4","Type":"ContainerDied","Data":"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"}
Oct 30 20:19:08.029465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.029274    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr" event={"ID":"659ecac3-50cb-469c-954a-3320295ea4b4","Type":"ContainerDied","Data":"36bc6424c1702cae45185b65c785c1660bed2a1921d4194ef4ad801247cf5198"}
Oct 30 20:19:08.029465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.029296    5085 scope.go:122] "RemoveContainer" containerID="acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"
Oct 30 20:19:08.040388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.040370    5085 scope.go:122] "RemoveContainer" containerID="acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"
Oct 30 20:19:08.040677 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:08.040657    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94\": container with ID starting with acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94 not found: ID does not exist" containerID="acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"
Oct 30 20:19:08.040763 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.040685    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94"} err="failed to get container status \"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94\": rpc error: code = NotFound desc = could not find container \"acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94\": container with ID starting with acdd192f32dac833446d738f836355e78a36e7f647dfdf9ca9981ef7a6e79c94 not found: ID does not exist"
Oct 30 20:19:08.187610 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.187536    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:19:08.196978 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.196941    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-0b9215-7646959cf5-4nmlr"]
Oct 30 20:19:08.632779 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:08.632749    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659ecac3-50cb-469c-954a-3320295ea4b4" path="/var/lib/kubelet/pods/659ecac3-50cb-469c-954a-3320295ea4b4/volumes"
Oct 30 20:19:14.335677 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:14.335638    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:19:18.296946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:18.296913    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:19:18.297369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:18.297117    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" containerID="cri-o://f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303" gracePeriod=30
Oct 30 20:19:19.505718 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:19.505679    5085 patch_prober.go:29] interesting pod/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:19:19.506183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:19.505740    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:19:21.939306 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:21.939267    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:19:23.565234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.565173    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:19:23.695746 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.695714    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.695945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.695760    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.695945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.695806    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.695945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.695866    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.695945 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.695941    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.696203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696037    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.696203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696068    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.696203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696141    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.696393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696230    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\") pod \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\" (UID: \"aab56326-d5ac-497b-a4f5-b9b5af4f4d38\") "
Oct 30 20:19:23.696393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696235    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:23.696393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696259    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:23.697108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.696597    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:23.697108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.697008    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:23.697108 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.697102    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.697356 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.697124    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.697356 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.697124    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:23.697356 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.697140    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.698194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.698139    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:19:23.698368 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.698346    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:23.698445 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.698415    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "kube-api-access-l6kjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:23.698599 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.698556    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy" pod "aab56326-d5ac-497b-a4f5-b9b5af4f4d38" (UID: "aab56326-d5ac-497b-a4f5-b9b5af4f4d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:19:23.797996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.797964    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.797996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.797993    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.797996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.798003    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6kjq\" (UniqueName: \"kubernetes.io/projected/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-kube-api-access-l6kjq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.798225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.798012    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.798225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.798022    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:23.798225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:23.798031    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aab56326-d5ac-497b-a4f5-b9b5af4f4d38-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:19:24.150687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.150649    5085 generic.go:356] "Generic (PLEG): container finished" podID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerID="f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303" exitCode=0
Oct 30 20:19:24.150906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.150727    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"
Oct 30 20:19:24.150906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.150731    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerDied","Data":"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"}
Oct 30 20:19:24.150906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.150791    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49" event={"ID":"aab56326-d5ac-497b-a4f5-b9b5af4f4d38","Type":"ContainerDied","Data":"b55fed1415a3d7ff92acfd42432b62d093ea7b6ec9c5ea8da0e72cbcf014afa6"}
Oct 30 20:19:24.150906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.150804    5085 scope.go:122] "RemoveContainer" containerID="f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"
Oct 30 20:19:24.162251 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.162233    5085 scope.go:122] "RemoveContainer" containerID="784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"
Oct 30 20:19:24.171760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.171742    5085 scope.go:122] "RemoveContainer" containerID="f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"
Oct 30 20:19:24.172003 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:24.171983    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303\": container with ID starting with f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303 not found: ID does not exist" containerID="f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"
Oct 30 20:19:24.172051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.172011    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303"} err="failed to get container status \"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303\": rpc error: code = NotFound desc = could not find container \"f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303\": container with ID starting with f2d9d658523a10229c1aebf6acd7af0b34b06081e7f52c05b32a8c4f8d16e303 not found: ID does not exist"
Oct 30 20:19:24.172051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.172028    5085 scope.go:122] "RemoveContainer" containerID="784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"
Oct 30 20:19:24.172242 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:24.172227    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1\": container with ID starting with 784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1 not found: ID does not exist" containerID="784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"
Oct 30 20:19:24.172282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.172248    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1"} err="failed to get container status \"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1\": rpc error: code = NotFound desc = could not find container \"784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1\": container with ID starting with 784ff5239769620e4eaa08a9427bcde051aee556b961ae6f63f1f96eda8d47b1 not found: ID does not exist"
Oct 30 20:19:24.310430 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.310395    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:19:24.357276 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.357211    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-0b9215-openshift-default-794f8d99c7-bxs49"]
Oct 30 20:19:24.632780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:24.632743    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" path="/var/lib/kubelet/pods/aab56326-d5ac-497b-a4f5-b9b5af4f4d38/volumes"
Oct 30 20:19:28.315462 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.315382    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:19:28.316074 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316060    5085 cpu_manager.go:414] "containerMap: removing container" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316075    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316086    5085 cpu_manager.go:414] "containerMap: removing container" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316092    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316116    5085 cpu_manager.go:414] "containerMap: removing container" podUID="659ecac3-50cb-469c-954a-3320295ea4b4" containerName="maas-api"
Oct 30 20:19:28.316134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316122    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659ecac3-50cb-469c-954a-3320295ea4b4" containerName="maas-api"
Oct 30 20:19:28.316303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316219    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316227    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="aab56326-d5ac-497b-a4f5-b9b5af4f4d38" containerName="istio-proxy"
Oct 30 20:19:28.316303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.316237    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="659ecac3-50cb-469c-954a-3320295ea4b4" containerName="maas-api"
Oct 30 20:19:28.321458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.321438    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.354441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.354409    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.354576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.354524    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tp9l\" (UniqueName: \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.354576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.354558    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.354686 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.354639    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.428997 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.428967    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-fe49e7-serving-cert\""
Oct 30 20:19:28.433138 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.433112    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:19:28.456358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.456329    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.456524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.456417    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tp9l\" (UniqueName: \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.456524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.456451    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.456524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.456490    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.456737 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:28.456592    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-serving-cert: secret "maas-api-e2e-shared-b-w3-fe49e7-serving-cert" not found
Oct 30 20:19:28.456737 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:28.456664    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls podName:0b5b87b5-dec5-43f2-8d44-e595f3964ad1 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:28.956644269 +0000 UTC m=+3278.461048020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls") pod "maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" (UID: "0b5b87b5-dec5-43f2-8d44-e595f3964ad1") : secret "maas-api-e2e-shared-b-w3-fe49e7-serving-cert" not found
Oct 30 20:19:28.457074 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.457053    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.459095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.459069    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.515686 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.515653    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tp9l\" (UniqueName: \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.961748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.961708    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:28.964143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:28.964110    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:29.237536 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:29.237457    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:29.615807 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:19:29.615776    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5b87b5_dec5_43f2_8d44_e595f3964ad1.slice/crio-159223853caa038727ada49818e5ca5eb94d666c080ca47229b21a5b90aeb8aa WatchSource:0}: Error finding container 159223853caa038727ada49818e5ca5eb94d666c080ca47229b21a5b90aeb8aa: Status 404 returned error can't find the container with id 159223853caa038727ada49818e5ca5eb94d666c080ca47229b21a5b90aeb8aa
Oct 30 20:19:29.616243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:29.615969    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:19:30.197759 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:30.197726    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" event={"ID":"0b5b87b5-dec5-43f2-8d44-e595f3964ad1","Type":"ContainerStarted","Data":"159223853caa038727ada49818e5ca5eb94d666c080ca47229b21a5b90aeb8aa"}
Oct 30 20:19:31.206888 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:31.206845    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" event={"ID":"0b5b87b5-dec5-43f2-8d44-e595f3964ad1","Type":"ContainerStarted","Data":"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"}
Oct 30 20:19:31.207336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:31.206990    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:31.328220 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:31.328168    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" podStartSLOduration=2.481730287 podStartE2EDuration="3.328152561s" totalImagesPullingTime="846.422274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:31.326000097 +0000 UTC m=+3280.830403846" watchObservedRunningTime="2026-10-30 20:19:31.328152561 +0000 UTC m=+3280.832556307"
Oct 30 20:19:37.222086 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:37.222056    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:19:43.122254 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.122220    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:19:43.127953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.127932    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.224756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.224720    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-b37684-serving-cert\""
Oct 30 20:19:43.228832 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.228807    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:19:43.235482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.235459    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhghn\" (UniqueName: \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.235554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.235489    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.235632 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.235591    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.235752 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.235735    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.336878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.336845    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.337080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.336966    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhghn\" (UniqueName: \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.337080 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:43.336988    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-serving-cert: secret "maas-api-e2e-shared-b-w1-b37684-serving-cert" not found
Oct 30 20:19:43.337080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.336996    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.337080 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:43.337059    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls podName:f1022e3f-ea50-4bca-961f-e8a71e390086 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:43.837039151 +0000 UTC m=+3293.341442891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls") pod "maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" (UID: "f1022e3f-ea50-4bca-961f-e8a71e390086") : secret "maas-api-e2e-shared-b-w1-b37684-serving-cert" not found
Oct 30 20:19:43.337312 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.337096    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.337926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.337896    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.339472 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.339450    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.400905 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.400838    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhghn\" (UniqueName: \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.843257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.843220    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:43.845712 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:43.845685    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:44.042087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:44.042056    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:44.417192 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:19:44.417149    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1022e3f_ea50_4bca_961f_e8a71e390086.slice/crio-ce995be8755231bc61bf6eb138b272ac7140f49dbbbaa6b19bcc674862583ac0 WatchSource:0}: Error finding container ce995be8755231bc61bf6eb138b272ac7140f49dbbbaa6b19bcc674862583ac0: Status 404 returned error can't find the container with id ce995be8755231bc61bf6eb138b272ac7140f49dbbbaa6b19bcc674862583ac0
Oct 30 20:19:44.418106 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:44.418072    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:19:45.333046 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:45.333013    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" event={"ID":"f1022e3f-ea50-4bca-961f-e8a71e390086","Type":"ContainerStarted","Data":"ce995be8755231bc61bf6eb138b272ac7140f49dbbbaa6b19bcc674862583ac0"}
Oct 30 20:19:46.136030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:46.135990    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:19:46.350547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:46.350507    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" event={"ID":"f1022e3f-ea50-4bca-961f-e8a71e390086","Type":"ContainerStarted","Data":"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"}
Oct 30 20:19:46.350756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:46.350677    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:46.484069 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:46.484017    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" podStartSLOduration=2.598119137 podStartE2EDuration="3.484003222s" totalImagesPullingTime="885.884085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:46.47925118 +0000 UTC m=+3295.983654927" watchObservedRunningTime="2026-10-30 20:19:46.484003222 +0000 UTC m=+3295.988406968"
Oct 30 20:19:51.428799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:51.428757    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:19:51.445847 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:51.445824    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:19:51.939368 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:51.939333    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:19:52.364996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:52.364910    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:19:58.529951 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.529916    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:19:58.535918 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.535891    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.585730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.585700    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-64120b-serving-cert\""
Oct 30 20:19:58.627300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.627270    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.627487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.627324    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtv6x\" (UniqueName: \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.627487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.627442    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.627647 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.627510    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.636377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.636344    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:19:58.728907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.728876    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.729078 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.728974    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.729146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.729129    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.729201 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.729182    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtv6x\" (UniqueName: \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.729302 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:58.729282    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-serving-cert: secret "maas-api-e2e-shared-b-w2-64120b-serving-cert" not found
Oct 30 20:19:58.729371 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:19:58.729359    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls podName:94420e6d-c47e-4be9-9b91-513eaf86bc29 nodeName:}" failed. No retries permitted until 2026-10-30 20:19:59.229337062 +0000 UTC m=+3308.733740804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls") pod "maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" (UID: "94420e6d-c47e-4be9-9b91-513eaf86bc29") : secret "maas-api-e2e-shared-b-w2-64120b-serving-cert" not found
Oct 30 20:19:58.729769 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.729744    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.731437 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.731414    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:58.789840 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:58.789760    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtv6x\" (UniqueName: \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:59.238590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:59.238538    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:59.240931 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:59.240908    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:59.449756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:59.449723    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:19:59.823987 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:19:59.823952    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:19:59.824589 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:19:59.824541    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94420e6d_c47e_4be9_9b91_513eaf86bc29.slice/crio-6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042 WatchSource:0}: Error finding container 6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042: Status 404 returned error can't find the container with id 6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042
Oct 30 20:20:00.460942 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:00.460904    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" event={"ID":"94420e6d-c47e-4be9-9b91-513eaf86bc29","Type":"ContainerStarted","Data":"6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042"}
Oct 30 20:20:01.469644 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:01.469609    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" event={"ID":"94420e6d-c47e-4be9-9b91-513eaf86bc29","Type":"ContainerStarted","Data":"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"}
Oct 30 20:20:01.470080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:01.469828    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:20:01.584285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:01.584234    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" podStartSLOduration=2.6502183070000003 podStartE2EDuration="3.58422142s" totalImagesPullingTime="934.003113ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:01.581169273 +0000 UTC m=+3311.085573019" watchObservedRunningTime="2026-10-30 20:20:01.58422142 +0000 UTC m=+3311.088625169"
Oct 30 20:20:07.484699 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:07.484669    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:20:15.057903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:15.057858    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"]
Oct 30 20:20:15.097940 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:15.097906    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a53b2440-dsqc7"]
Oct 30 20:20:15.857013 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:15.856979    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:20:16.633064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:16.633025    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39bc542-fea9-4022-a035-9af4808863ff" path="/var/lib/kubelet/pods/b39bc542-fea9-4022-a035-9af4808863ff/volumes"
Oct 30 20:20:21.939201 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:20:21.939169    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:20:24.051298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:24.051263    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"]
Oct 30 20:20:24.097902 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:24.097867    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33a8ab3a-njwjl"]
Oct 30 20:20:24.632481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:24.632449    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ec4cd7-d15d-4897-aa9d-97d21cf13465" path="/var/lib/kubelet/pods/e3ec4cd7-d15d-4897-aa9d-97d21cf13465/volumes"
Oct 30 20:20:40.053495 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:40.053461    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"]
Oct 30 20:20:40.099946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:40.099909    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61d18d9a-5zq7w"]
Oct 30 20:20:40.632528 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:40.632498    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a2ad73f-9b90-4cd8-b37e-1d691856898a" path="/var/lib/kubelet/pods/6a2ad73f-9b90-4cd8-b37e-1d691856898a/volumes"
Oct 30 20:20:51.048373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.048336    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"]
Oct 30 20:20:51.094901 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.094872    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b06305b2-x2469"]
Oct 30 20:20:51.824834 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.824800    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:20:51.829222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.829206    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:51.881293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.881256    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-4b0bf3-serving-cert\""
Oct 30 20:20:51.886290 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.886258    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:20:51.915369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.915335    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjzzb\" (UniqueName: \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:51.915552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.915392    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:51.915552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.915544    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:51.915712 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:51.915662    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:51.940850 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:20:51.940809    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:20:52.016635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.016585    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.016820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.016695    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjzzb\" (UniqueName: \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.016820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.016732    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.016944 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.016847    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.017018 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:20:52.016937    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-serving-cert: secret "maas-api-e2e-explicit-4b0bf3-serving-cert" not found
Oct 30 20:20:52.017084 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:20:52.017036    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls podName:94deef26-fccd-431c-8cd6-501656473003 nodeName:}" failed. No retries permitted until 2026-10-30 20:20:52.517012728 +0000 UTC m=+3362.021416457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls") pod "maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" (UID: "94deef26-fccd-431c-8cd6-501656473003") : secret "maas-api-e2e-explicit-4b0bf3-serving-cert" not found
Oct 30 20:20:52.017182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.017161    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.019198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.019181    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.076247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.076158    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjzzb\" (UniqueName: \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.522698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.522659    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.525510 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.525481    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") pod \"maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:52.631893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.631860    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc2aab29-1908-4cae-b468-434f669aa4d7" path="/var/lib/kubelet/pods/cc2aab29-1908-4cae-b468-434f669aa4d7/volumes"
Oct 30 20:20:52.743395 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:52.743362    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:53.119324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:53.119292    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:20:53.119783 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:20:53.119702    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94deef26_fccd_431c_8cd6_501656473003.slice/crio-50155add417ee85fe1d8412b66d79968f26c0e65b51e9351e5d216b519c2f9e2 WatchSource:0}: Error finding container 50155add417ee85fe1d8412b66d79968f26c0e65b51e9351e5d216b519c2f9e2: Status 404 returned error can't find the container with id 50155add417ee85fe1d8412b66d79968f26c0e65b51e9351e5d216b519c2f9e2
Oct 30 20:20:53.866373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:53.866332    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" event={"ID":"94deef26-fccd-431c-8cd6-501656473003","Type":"ContainerStarted","Data":"50155add417ee85fe1d8412b66d79968f26c0e65b51e9351e5d216b519c2f9e2"}
Oct 30 20:20:54.297134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:54.297101    5085 scope.go:122] "RemoveContainer" containerID="dc042f5a2d5d33164f4935fb591fe9a6cb489c579e40feb8b98d448c9814f561"
Oct 30 20:20:54.875736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:54.875698    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" event={"ID":"94deef26-fccd-431c-8cd6-501656473003","Type":"ContainerStarted","Data":"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"}
Oct 30 20:20:54.875938 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:54.875886    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:20:54.988939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:54.988888    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" podStartSLOduration=3.170566632 podStartE2EDuration="3.988873768s" totalImagesPullingTime="818.307136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:54.987386944 +0000 UTC m=+3364.491790692" watchObservedRunningTime="2026-10-30 20:20:54.988873768 +0000 UTC m=+3364.493277516"
Oct 30 20:20:57.049677 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:57.049641    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"]
Oct 30 20:20:57.096203 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:57.096168    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-120790c2-84kmp"]
Oct 30 20:20:58.633209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:20:58.633110    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c55af63-bb1c-451b-929e-0f628245acdf" path="/var/lib/kubelet/pods/6c55af63-bb1c-451b-929e-0f628245acdf/volumes"
Oct 30 20:21:00.454494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:00.454459    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:21:00.888926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:00.888848    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:21:01.056134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:01.056102    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"]
Oct 30 20:21:01.059865 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:01.059839    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff31619-zbtfc"]
Oct 30 20:21:02.632493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:02.632463    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d92a109a-47e5-4562-a344-63cc429f826d" path="/var/lib/kubelet/pods/d92a109a-47e5-4562-a344-63cc429f826d/volumes"
Oct 30 20:21:07.049162 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:07.049131    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"]
Oct 30 20:21:07.095668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:07.095633    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2f60440-mv9rt"]
Oct 30 20:21:08.632506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:08.632472    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30fdcdb-b163-43d2-98d8-454ae19d6c71" path="/var/lib/kubelet/pods/d30fdcdb-b163-43d2-98d8-454ae19d6c71/volumes"
Oct 30 20:21:12.049786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:12.049748    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"]
Oct 30 20:21:12.096833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:12.096803    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95d00d90-bnmhz"]
Oct 30 20:21:12.632547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:12.632511    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ae12b7-75c0-481b-8c50-64f16b09d1c5" path="/var/lib/kubelet/pods/34ae12b7-75c0-481b-8c50-64f16b09d1c5/volumes"
Oct 30 20:21:21.939927 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:21:21.939891    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:21:29.055189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:29.055155    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"]
Oct 30 20:21:29.101914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:29.101882    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7120eee0-jr66p"]
Oct 30 20:21:30.634743 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:30.634709    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13402d8-bdcc-49f4-bf02-0ef0b477fdae" path="/var/lib/kubelet/pods/e13402d8-bdcc-49f4-bf02-0ef0b477fdae/volumes"
Oct 30 20:21:36.151369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:36.151331    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:21:51.939745 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:21:51.939708    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:21:54.326291 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:54.326259    5085 scope.go:122] "RemoveContainer" containerID="c1d353b4a2864d24e0af7b78f8a208472b90d4eee87bf411d9ac308cecb13e50"
Oct 30 20:21:54.337546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:54.337520    5085 scope.go:122] "RemoveContainer" containerID="1b8f49463511ebd2cc956b8a34bcf34c524d210832b07576fc2864993634e0c2"
Oct 30 20:21:54.347949 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:54.347928    5085 scope.go:122] "RemoveContainer" containerID="ae9ebdbdd69d5c1794949e989417505a77d7d3fd301a2d8788470f2ef6020b88"
Oct 30 20:21:54.357217 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:54.357194    5085 scope.go:122] "RemoveContainer" containerID="1453aa4a77e3d212c2ccff8ac44d9e9f0700f86f44ca0f0c1fef8e4110d26ba2"
Oct 30 20:21:54.366132 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:21:54.366108    5085 scope.go:122] "RemoveContainer" containerID="5b32b2e800eaf731bc5c76e9fe6f7c6855cced81ea21f7997d36b95f5d2ab2a8"
Oct 30 20:22:11.849775 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:11.849737    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:22:20.687611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:20.687578    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:22:20.688039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:20.687799    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy" containerID="cri-o://e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f" gracePeriod=30
Oct 30 20:22:21.943913 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:22:21.943865    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:22:25.971000 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:25.970974    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:22:26.002117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002087    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002139    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002172    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002250    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002280    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002313    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002425    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002549    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002633    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002650    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002679    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\") pod \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\" (UID: \"06e0d6e4-7905-41e5-8d6d-c185ade9549d\") "
Oct 30 20:22:26.002760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002733    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:26.003123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.002875    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:26.003181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003133    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:26.003300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003282    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.003374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003306    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.003374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003325    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.003374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003341    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.003374 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.003356    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.004439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.004405    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:26.004659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.004641    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:26.005111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.005093    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "kube-api-access-4t8cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:26.005161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.005110    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy" pod "06e0d6e4-7905-41e5-8d6d-c185ade9549d" (UID: "06e0d6e4-7905-41e5-8d6d-c185ade9549d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:26.107886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.106492    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.107886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.106537    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.107886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.106551    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.107886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.106580    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t8cz\" (UniqueName: \"kubernetes.io/projected/06e0d6e4-7905-41e5-8d6d-c185ade9549d-kube-api-access-4t8cz\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:22:26.579476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.579443    5085 generic.go:356] "Generic (PLEG): container finished" podID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerID="e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f" exitCode=0
Oct 30 20:22:26.579680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.579509    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"
Oct 30 20:22:26.579680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.579541    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" event={"ID":"06e0d6e4-7905-41e5-8d6d-c185ade9549d","Type":"ContainerDied","Data":"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"}
Oct 30 20:22:26.579680 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.579650    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b" event={"ID":"06e0d6e4-7905-41e5-8d6d-c185ade9549d","Type":"ContainerDied","Data":"b5a5b6d0bda506d0bfe37042a7ae5016fab7f8688635141c0f2f10591507b409"}
Oct 30 20:22:26.579822 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.579685    5085 scope.go:122] "RemoveContainer" containerID="e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"
Oct 30 20:22:26.591105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.591087    5085 scope.go:122] "RemoveContainer" containerID="e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"
Oct 30 20:22:26.591379 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:22:26.591360    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f\": container with ID starting with e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f not found: ID does not exist" containerID="e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"
Oct 30 20:22:26.591457 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.591392    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f"} err="failed to get container status \"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f\": rpc error: code = NotFound desc = could not find container \"e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f\": container with ID starting with e764efd1a17c11c076a34e2039f1cfd9b4c6d73eda9dcba0f863e071158ae20f not found: ID does not exist"
Oct 30 20:22:26.741672 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.741639    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:22:26.748729 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:26.748707    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-5m89b"]
Oct 30 20:22:28.632843 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:28.632769    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" path="/var/lib/kubelet/pods/06e0d6e4-7905-41e5-8d6d-c185ade9549d/volumes"
Oct 30 20:22:49.905277 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:49.905240    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:22:51.939423 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:22:51.939381    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:22:54.424163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:54.424131    5085 scope.go:122] "RemoveContainer" containerID="c5c4d825ef78e677a12c454a255a1a81c8682189655f1542b39e73606c27b283"
Oct 30 20:22:54.434500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:54.434481    5085 scope.go:122] "RemoveContainer" containerID="420b278a567ae28e4d18eb7329782d0095a9929d4b7a42c47548bfb25ebd24eb"
Oct 30 20:22:54.444259 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:54.444240    5085 scope.go:122] "RemoveContainer" containerID="c95b86ae2c8f166b0fd7002ac0c8731b95f41ac26bcf2ba03e06ba65b49e6ec4"
Oct 30 20:22:54.456120 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:22:54.456095    5085 scope.go:122] "RemoveContainer" containerID="289d8ac144f4a3a1ca9b2ee57589d8968026ecb943ddbcd385d9912b01ce9be2"
Oct 30 20:23:21.939435 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:23:21.939398    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:23:25.278180 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:23:25.278139    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:23:51.940324 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:23:51.940286    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:24:17.377215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:17.377182    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:24:17.377692 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:17.377380    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" containerID="cri-o://7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d" gracePeriod=30
Oct 30 20:24:20.910867 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:20.910828    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:24:20.911287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:20.911100    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" podUID="94deef26-fccd-431c-8cd6-501656473003" containerName="maas-api" containerID="cri-o://06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d" gracePeriod=30
Oct 30 20:24:21.150743 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.150672    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:24:21.206900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.206871    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:24:21.334793 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.334758    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\") pod \"94deef26-fccd-431c-8cd6-501656473003\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") "
Oct 30 20:24:21.335089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.335069    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") pod \"94deef26-fccd-431c-8cd6-501656473003\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") "
Oct 30 20:24:21.335253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.335240    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\" (UniqueName: \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\") pod \"94deef26-fccd-431c-8cd6-501656473003\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") "
Oct 30 20:24:21.335416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.335403    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\") pod \"94deef26-fccd-431c-8cd6-501656473003\" (UID: \"94deef26-fccd-431c-8cd6-501656473003\") "
Oct 30 20:24:21.335549 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.335520    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca" pod "94deef26-fccd-431c-8cd6-501656473003" (UID: "94deef26-fccd-431c-8cd6-501656473003"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:21.336088 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.336068    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94deef26-fccd-431c-8cd6-501656473003-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:21.338976 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.338951    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs" pod "94deef26-fccd-431c-8cd6-501656473003" (UID: "94deef26-fccd-431c-8cd6-501656473003"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:21.341516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.341478    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb" pod "94deef26-fccd-431c-8cd6-501656473003" (UID: "94deef26-fccd-431c-8cd6-501656473003"). InnerVolumeSpecName "kube-api-access-tjzzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:21.342378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.341733    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls" pod "94deef26-fccd-431c-8cd6-501656473003" (UID: "94deef26-fccd-431c-8cd6-501656473003"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:21.389656 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.389613    5085 generic.go:356] "Generic (PLEG): container finished" podID="94deef26-fccd-431c-8cd6-501656473003" containerID="06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d" exitCode=0
Oct 30 20:24:21.389831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.389656    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"
Oct 30 20:24:21.389831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.389675    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" event={"ID":"94deef26-fccd-431c-8cd6-501656473003","Type":"ContainerDied","Data":"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"}
Oct 30 20:24:21.389831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.389734    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj" event={"ID":"94deef26-fccd-431c-8cd6-501656473003","Type":"ContainerDied","Data":"50155add417ee85fe1d8412b66d79968f26c0e65b51e9351e5d216b519c2f9e2"}
Oct 30 20:24:21.389831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.389765    5085 scope.go:122] "RemoveContainer" containerID="06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"
Oct 30 20:24:21.402275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.402256    5085 scope.go:122] "RemoveContainer" containerID="06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"
Oct 30 20:24:21.402556 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:24:21.402539    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d\": container with ID starting with 06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d not found: ID does not exist" containerID="06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"
Oct 30 20:24:21.402556 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.402576    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d"} err="failed to get container status \"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d\": rpc error: code = NotFound desc = could not find container \"06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d\": container with ID starting with 06a5811172979b628f9138b2bc23d853c62125834ca17babb11fd759d487ac9d not found: ID does not exist"
Oct 30 20:24:21.437621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.437513    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:21.437621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.437541    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjzzb\" (UniqueName: \"kubernetes.io/projected/94deef26-fccd-431c-8cd6-501656473003-kube-api-access-tjzzb\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:21.437621 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.437552    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94deef26-fccd-431c-8cd6-501656473003-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:21.569044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.569010    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:24:21.574558 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:21.574531    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-4b0bf3-6986bb5999-894gj"]
Oct 30 20:24:21.941529 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:24:21.941487    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:24:22.247836 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.247755    5085 patch_prober.go:29] interesting pod/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:24:22.247994 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.247823    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:24:22.633032 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.632997    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94deef26-fccd-431c-8cd6-501656473003" path="/var/lib/kubelet/pods/94deef26-fccd-431c-8cd6-501656473003/volumes"
Oct 30 20:24:22.677920 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.677896    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:24:22.763489 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763458    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.763820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763505    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.763820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763546    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.763820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763614    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.763820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763743    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.763820 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763808    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763830    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763865    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763917    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.763996    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764003    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764051    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\") pod \"eba37489-7b14-4b66-b0c5-9778066b65c6\" (UID: \"eba37489-7b14-4b66-b0c5-9778066b65c6\") "
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764307    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:22.764466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764334    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:22.764928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764764    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eba37489-7b14-4b66-b0c5-9778066b65c6-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.764928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764785    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.764928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764802    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.764928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764816    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.764928 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.764829    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.766082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.766061    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "kube-api-access-k7spk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:22.766393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.766362    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:24:22.766393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.766373    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:22.766532 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.766391    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy" pod "eba37489-7b14-4b66-b0c5-9778066b65c6" (UID: "eba37489-7b14-4b66-b0c5-9778066b65c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:24:22.866387 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.866287    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.866387 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.866320    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.866387 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.866329    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eba37489-7b14-4b66-b0c5-9778066b65c6-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:22.866387 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:22.866339    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7spk\" (UniqueName: \"kubernetes.io/projected/eba37489-7b14-4b66-b0c5-9778066b65c6-kube-api-access-k7spk\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:24:23.409605 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.409551    5085 generic.go:356] "Generic (PLEG): container finished" podID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerID="7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d" exitCode=0
Oct 30 20:24:23.410023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.409625    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"
Oct 30 20:24:23.410023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.409639    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" event={"ID":"eba37489-7b14-4b66-b0c5-9778066b65c6","Type":"ContainerDied","Data":"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"}
Oct 30 20:24:23.410023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.409707    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc" event={"ID":"eba37489-7b14-4b66-b0c5-9778066b65c6","Type":"ContainerDied","Data":"713523ace9d5260f75b473bd3a3c3862364a18821fc709f99cb9cb459c969474"}
Oct 30 20:24:23.410023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.409737    5085 scope.go:122] "RemoveContainer" containerID="7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"
Oct 30 20:24:23.420479 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.420457    5085 scope.go:122] "RemoveContainer" containerID="7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"
Oct 30 20:24:23.420770 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:24:23.420751    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d\": container with ID starting with 7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d not found: ID does not exist" containerID="7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"
Oct 30 20:24:23.420842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.420783    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d"} err="failed to get container status \"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d\": rpc error: code = NotFound desc = could not find container \"7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d\": container with ID starting with 7520ac1a40d6c85dce98a432922e38e2096c6137abfa8dd8721f9341d89d666d not found: ID does not exist"
Oct 30 20:24:23.567696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.567660    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:24:23.614843 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:23.614805    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-jdpjc"]
Oct 30 20:24:24.632022 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:24.631984    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" path="/var/lib/kubelet/pods/eba37489-7b14-4b66-b0c5-9778066b65c6/volumes"
Oct 30 20:24:51.478171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:51.478140    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:24:51.497030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:24:51.497006    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:24:51.939029 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:24:51.938984    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:25:21.939462 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:25:21.939425    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:25:24.743614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:24.743582    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:25:35.736841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:35.736759    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:25:35.737371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:35.736971    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" containerID="cri-o://a8dbf9c385776a7aada6f19ff6587905f8716d2fa6e9dccfcfed1b932448a3bc" gracePeriod=30
Oct 30 20:25:37.179145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:37.179114    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:25:37.179575 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:37.179339    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" containerID="cri-o://9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95" gracePeriod=30
Oct 30 20:25:39.569637 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:39.569604    5085 patch_prober.go:29] interesting pod/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:25:39.570034 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:39.569665    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:25:40.981353 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:40.981319    5085 generic.go:356] "Generic (PLEG): container finished" podID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerID="a8dbf9c385776a7aada6f19ff6587905f8716d2fa6e9dccfcfed1b932448a3bc" exitCode=0
Oct 30 20:25:40.981816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:40.981398    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" event={"ID":"67737ccd-a64c-4995-aac6-d7f13ba3b93a","Type":"ContainerDied","Data":"a8dbf9c385776a7aada6f19ff6587905f8716d2fa6e9dccfcfed1b932448a3bc"}
Oct 30 20:25:41.013818 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.013762    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:25:41.106247 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106216    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106257    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106324    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106507    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106603 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106555    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106613    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.106739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106648    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:41.106739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106661    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.107182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.107120    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:41.107303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.107262    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.107303 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.107302    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\") pod \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\" (UID: \"67737ccd-a64c-4995-aac6-d7f13ba3b93a\") "
Oct 30 20:25:41.107850 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.107822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:41.108857 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.108822    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:41.108857 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.107985    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:41.109030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.106797    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109628    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109654    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109669    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109691    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109705    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109719    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109955    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:41.110609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.109984    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:41.126608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.124619    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p" pod "67737ccd-a64c-4995-aac6-d7f13ba3b93a" (UID: "67737ccd-a64c-4995-aac6-d7f13ba3b93a"). InnerVolumeSpecName "kube-api-access-77g8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:41.211174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.211127    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.211174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.211161    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77g8p\" (UniqueName: \"kubernetes.io/projected/67737ccd-a64c-4995-aac6-d7f13ba3b93a-kube-api-access-77g8p\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.211174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.211172    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67737ccd-a64c-4995-aac6-d7f13ba3b93a-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:41.448473 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.448436    5085 patch_prober.go:29] interesting pod/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:25:41.448700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.448491    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:25:41.990129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.990101    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"
Oct 30 20:25:41.990601 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.990136    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7" event={"ID":"67737ccd-a64c-4995-aac6-d7f13ba3b93a","Type":"ContainerDied","Data":"d3ec974c3b5af24a5e29117787060e0fca66f3a2db1bfa8e1dc07f530e4b368f"}
Oct 30 20:25:41.990601 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:41.990200    5085 scope.go:122] "RemoveContainer" containerID="a8dbf9c385776a7aada6f19ff6587905f8716d2fa6e9dccfcfed1b932448a3bc"
Oct 30 20:25:42.195926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.195897    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:25:42.242766 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.242702    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bbfa07-openshift-default-77bf654b75-w5pw7"]
Oct 30 20:25:42.454732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.454708    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:25:42.523347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523270    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.523347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523339    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.523629 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523360    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.523629 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523382    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.523629 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523414    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.523804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523630    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:42.523804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523731    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:42.523911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523810    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:42.523911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523899    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:42.523989 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.523942    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.524052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524022    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.524135 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524116    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.524307 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524235    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\") pod \"82777244-3c44-4370-bb57-3126b5777e2e\" (UID: \"82777244-3c44-4370-bb57-3126b5777e2e\") "
Oct 30 20:25:42.524408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524364    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:42.524841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524808    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.524841 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524834    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.525037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524849    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.525037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524865    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.525037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.524880    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82777244-3c44-4370-bb57-3126b5777e2e-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.526152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.526120    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "kube-api-access-7dvxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:42.526265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.526152    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:42.526619 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.526592    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:25:42.526765 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.526705    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo" pod "82777244-3c44-4370-bb57-3126b5777e2e" (UID: "82777244-3c44-4370-bb57-3126b5777e2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:25:42.625493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.625462    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82777244-3c44-4370-bb57-3126b5777e2e-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.625493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.625489    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.625493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.625498    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvxj\" (UniqueName: \"kubernetes.io/projected/82777244-3c44-4370-bb57-3126b5777e2e-kube-api-access-7dvxj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.625737 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.625508    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82777244-3c44-4370-bb57-3126b5777e2e-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:25:42.632388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.632361    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" path="/var/lib/kubelet/pods/67737ccd-a64c-4995-aac6-d7f13ba3b93a/volumes"
Oct 30 20:25:42.998849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.998809    5085 generic.go:356] "Generic (PLEG): container finished" podID="82777244-3c44-4370-bb57-3126b5777e2e" containerID="9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95" exitCode=0
Oct 30 20:25:42.999255 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.998869    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"
Oct 30 20:25:42.999255 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.998895    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" event={"ID":"82777244-3c44-4370-bb57-3126b5777e2e","Type":"ContainerDied","Data":"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"}
Oct 30 20:25:42.999255 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.998947    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc" event={"ID":"82777244-3c44-4370-bb57-3126b5777e2e","Type":"ContainerDied","Data":"b4253dabafba7aaed7543ee2007876bd3dd4bd3e3dcbe2637f204f74d82b975c"}
Oct 30 20:25:42.999255 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:42.998973    5085 scope.go:122] "RemoveContainer" containerID="9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"
Oct 30 20:25:43.009518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:43.009489    5085 scope.go:122] "RemoveContainer" containerID="9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"
Oct 30 20:25:43.009822 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:25:43.009799    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95\": container with ID starting with 9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95 not found: ID does not exist" containerID="9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"
Oct 30 20:25:43.009890 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:43.009831    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95"} err="failed to get container status \"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95\": rpc error: code = NotFound desc = could not find container \"9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95\": container with ID starting with 9933b3dbf9549aa35df640b004096660b8145a10fbf17be103bd8f3614604a95 not found: ID does not exist"
Oct 30 20:25:43.159686 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:43.159655    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:25:43.205673 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:43.205640    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-s9ccc"]
Oct 30 20:25:44.632484 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:25:44.632448    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82777244-3c44-4370-bb57-3126b5777e2e" path="/var/lib/kubelet/pods/82777244-3c44-4370-bb57-3126b5777e2e/volumes"
Oct 30 20:25:51.940520 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:25:51.940485    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:26:21.939709 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:21.939669    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:26:24.209771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:24.209720    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:26:35.700419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.700384    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-952641"]
Oct 30 20:26:35.701082 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701066    5085 cpu_manager.go:414] "containerMap: removing container" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701082    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701101    5085 cpu_manager.go:414] "containerMap: removing container" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701110    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701121    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94deef26-fccd-431c-8cd6-501656473003" containerName="maas-api"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701126    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94deef26-fccd-431c-8cd6-501656473003" containerName="maas-api"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701141    5085 cpu_manager.go:414] "containerMap: removing container" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy"
Oct 30 20:26:35.701150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701146    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701156    5085 cpu_manager.go:414] "containerMap: removing container" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701161    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701260    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="82777244-3c44-4370-bb57-3126b5777e2e" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701279    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="06e0d6e4-7905-41e5-8d6d-c185ade9549d" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701286    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="67737ccd-a64c-4995-aac6-d7f13ba3b93a" containerName="istio-proxy"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701291    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94deef26-fccd-431c-8cd6-501656473003" containerName="maas-api"
Oct 30 20:26:35.701507 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.701299    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba37489-7b14-4b66-b0c5-9778066b65c6" containerName="istio-proxy"
Oct 30 20:26:35.705112 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.705092    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:35.750499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.750470    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5j2t\" (UniqueName: \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\") pod \"test-curl-7337-952641\" (UID: \"0002482e-6a8e-4989-9d0d-c47595a07213\") " pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:35.756066 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.756044    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-952641"]
Oct 30 20:26:35.798907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.798880    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\""
Oct 30 20:26:35.852169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.852139    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5j2t\" (UniqueName: \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\") pod \"test-curl-7337-952641\" (UID: \"0002482e-6a8e-4989-9d0d-c47595a07213\") " pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:35.911105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:35.911068    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5j2t\" (UniqueName: \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\") pod \"test-curl-7337-952641\" (UID: \"0002482e-6a8e-4989-9d0d-c47595a07213\") " pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:36.018166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:36.018097    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:36.395351 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:36.395319    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-952641"]
Oct 30 20:26:36.396469 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:26:36.396443    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0002482e_6a8e_4989_9d0d_c47595a07213.slice/crio-ba0c32d71c341b9c031fde833e1b4ed190f6db196588b5ccc316eb439ad706f9 WatchSource:0}: Error finding container ba0c32d71c341b9c031fde833e1b4ed190f6db196588b5ccc316eb439ad706f9: Status 404 returned error can't find the container with id ba0c32d71c341b9c031fde833e1b4ed190f6db196588b5ccc316eb439ad706f9
Oct 30 20:26:37.378010 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:37.377978    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-952641" event={"ID":"0002482e-6a8e-4989-9d0d-c47595a07213","Type":"ContainerStarted","Data":"58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e"}
Oct 30 20:26:37.378010 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:37.378018    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-952641" event={"ID":"0002482e-6a8e-4989-9d0d-c47595a07213","Type":"ContainerStarted","Data":"ba0c32d71c341b9c031fde833e1b4ed190f6db196588b5ccc316eb439ad706f9"}
Oct 30 20:26:37.487831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:37.487780    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7337-952641" podStartSLOduration=2.487764982 podStartE2EDuration="2.487764982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:26:37.48602995 +0000 UTC m=+3706.990433698" watchObservedRunningTime="2026-10-30 20:26:37.487764982 +0000 UTC m=+3706.992168731"
Oct 30 20:26:39.079799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:39.079763    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-952641"]
Oct 30 20:26:39.086600 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:39.086556    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-952641"]
Oct 30 20:26:39.391503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:39.391404    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7337-952641" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" containerName="test-curl-7337-952641" containerID="cri-o://58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e" gracePeriod=2
Oct 30 20:26:39.439764 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:39.439728    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-952641\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:40.675966 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:40.675925    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-952641\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:41.473876 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:41.473842    5085 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0002482e_6a8e_4989_9d0d_c47595a07213.slice/crio-conmon-58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:26:41.642631 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:41.642608    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:41.728805 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:41.728775    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\" (UniqueName: \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\") pod \"0002482e-6a8e-4989-9d0d-c47595a07213\" (UID: \"0002482e-6a8e-4989-9d0d-c47595a07213\") "
Oct 30 20:26:41.730933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:41.730908    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t" pod "0002482e-6a8e-4989-9d0d-c47595a07213" (UID: "0002482e-6a8e-4989-9d0d-c47595a07213"). InnerVolumeSpecName "kube-api-access-l5j2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:26:41.830590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:41.830532    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5j2t\" (UniqueName: \"kubernetes.io/projected/0002482e-6a8e-4989-9d0d-c47595a07213-kube-api-access-l5j2t\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:26:42.434285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.434241    5085 generic.go:356] "Generic (PLEG): container finished" podID="0002482e-6a8e-4989-9d0d-c47595a07213" containerID="58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e" exitCode=137
Oct 30 20:26:42.434285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.434286    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:42.434559 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.434347    5085 scope.go:122] "RemoveContainer" containerID="58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e"
Oct 30 20:26:42.444554 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.444537    5085 scope.go:122] "RemoveContainer" containerID="58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e"
Oct 30 20:26:42.444832 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:42.444812    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e\": container with ID starting with 58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e not found: ID does not exist" containerID="58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e"
Oct 30 20:26:42.444881 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.444843    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e"} err="failed to get container status \"58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e\": rpc error: code = NotFound desc = could not find container \"58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e\": container with ID starting with 58eadefe1b99e6123df7fc4b1f84495ce3ce429f4905450559ac38772e1a4c8e not found: ID does not exist"
Oct 30 20:26:42.490999 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:42.490960    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-952641\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" pod="opendatahub/test-curl-7337-952641"
Oct 30 20:26:42.632302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:26:42.632269    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" path="/var/lib/kubelet/pods/0002482e-6a8e-4989-9d0d-c47595a07213/volumes"
Oct 30 20:26:51.939631 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:26:51.939595    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:27:05.393003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:05.392966    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:27:21.939690 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:27:21.939655    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:27:50.789582 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.789533    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:27:50.790207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.790190    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" containerName="test-curl-7337-952641"
Oct 30 20:27:50.790207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.790206    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" containerName="test-curl-7337-952641"
Oct 30 20:27:50.790339 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.790334    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0002482e-6a8e-4989-9d0d-c47595a07213" containerName="test-curl-7337-952641"
Oct 30 20:27:50.793924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.793905    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.843911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.843880    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-742ff8-serving-cert\""
Oct 30 20:27:50.856539 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.856510    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.856698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.856550    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.856698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.856627    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pgg\" (UniqueName: \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.856773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.856719    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.895625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.895589    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:27:50.957584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.957539    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.957776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.957607    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.957776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.957646    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pgg\" (UniqueName: \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.957776 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.957719    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.957930 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:27:50.957872    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-serving-cert: secret "maas-api-e2e-shared-a-w4-742ff8-serving-cert" not found
Oct 30 20:27:50.957988 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:27:50.957936    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls podName:1212b686-9e45-4df9-832a-859013d89286 nodeName:}" failed. No retries permitted until 2026-10-30 20:27:51.457919097 +0000 UTC m=+3780.962322823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls") pod "maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" (UID: "1212b686-9e45-4df9-832a-859013d89286") : secret "maas-api-e2e-shared-a-w4-742ff8-serving-cert" not found
Oct 30 20:27:50.958366 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.958347    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:50.960132 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:50.960112    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:51.018269 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:51.018234    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pgg\" (UniqueName: \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:51.462999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:51.462964    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:51.465401 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:51.465377    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:51.712441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:51.712404    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:51.940077 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:27:51.940041    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:27:52.085526 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:52.085497    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:27:52.086776 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:27:52.086743    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1212b686_9e45_4df9_832a_859013d89286.slice/crio-28dd2f19b30b6c2ff621bc5ed3984e024e5c19d1745a04930b1ca8d6660a116e WatchSource:0}: Error finding container 28dd2f19b30b6c2ff621bc5ed3984e024e5c19d1745a04930b1ca8d6660a116e: Status 404 returned error can't find the container with id 28dd2f19b30b6c2ff621bc5ed3984e024e5c19d1745a04930b1ca8d6660a116e
Oct 30 20:27:52.088234 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:52.088216    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:27:52.928797 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:52.928762    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" event={"ID":"1212b686-9e45-4df9-832a-859013d89286","Type":"ContainerStarted","Data":"28dd2f19b30b6c2ff621bc5ed3984e024e5c19d1745a04930b1ca8d6660a116e"}
Oct 30 20:27:53.937868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:53.937836    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" event={"ID":"1212b686-9e45-4df9-832a-859013d89286","Type":"ContainerStarted","Data":"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"}
Oct 30 20:27:53.938268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:53.937890    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:54.053749 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:54.053697    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" podStartSLOduration=2.698836238 podStartE2EDuration="4.053683548s" totalImagesPullingTime="1.35484731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:54.049411178 +0000 UTC m=+3783.553814925" watchObservedRunningTime="2026-10-30 20:27:54.053683548 +0000 UTC m=+3783.558087295"
Oct 30 20:27:55.643075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.643044    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-768ecc"]
Oct 30 20:27:55.647369 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.647352    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:27:55.709076 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.708989    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-768ecc"]
Oct 30 20:27:55.718411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.718376    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttxg8\" (UniqueName: \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\") pod \"test-curl-7325-768ecc\" (UID: \"9d6c9120-2e45-490b-9f46-3418287c6079\") " pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:27:55.736827 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.736802    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\""
Oct 30 20:27:55.820308 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.820269    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttxg8\" (UniqueName: \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\") pod \"test-curl-7325-768ecc\" (UID: \"9d6c9120-2e45-490b-9f46-3418287c6079\") " pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:27:55.879058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.879023    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttxg8\" (UniqueName: \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\") pod \"test-curl-7325-768ecc\" (UID: \"9d6c9120-2e45-490b-9f46-3418287c6079\") " pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:27:55.959822 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:55.959794    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:27:56.329173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:56.329145    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-768ecc"]
Oct 30 20:27:56.330924 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:27:56.330897    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d6c9120_2e45_490b_9f46_3418287c6079.slice/crio-9148dd037b07fce724388a3567df535806ff870b78af2f0ddec15b502f9edec9 WatchSource:0}: Error finding container 9148dd037b07fce724388a3567df535806ff870b78af2f0ddec15b502f9edec9: Status 404 returned error can't find the container with id 9148dd037b07fce724388a3567df535806ff870b78af2f0ddec15b502f9edec9
Oct 30 20:27:56.958204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:56.958166    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-768ecc" event={"ID":"9d6c9120-2e45-490b-9f46-3418287c6079","Type":"ContainerStarted","Data":"06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74"}
Oct 30 20:27:56.958204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:56.958213    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-768ecc" event={"ID":"9d6c9120-2e45-490b-9f46-3418287c6079","Type":"ContainerStarted","Data":"9148dd037b07fce724388a3567df535806ff870b78af2f0ddec15b502f9edec9"}
Oct 30 20:27:57.071319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:57.071264    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-768ecc" podStartSLOduration=2.071251371 podStartE2EDuration="2.071251371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:57.069643127 +0000 UTC m=+3786.574046874" watchObservedRunningTime="2026-10-30 20:27:57.071251371 +0000 UTC m=+3786.575655111"
Oct 30 20:27:58.467726 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.467693    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-768ecc"]
Oct 30 20:27:58.473593 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.473543    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-768ecc"]
Oct 30 20:27:58.631970 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.631933    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-ff97ce"]
Oct 30 20:27:58.632742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.632717    5085 cpu_manager.go:414] "containerMap: removing container" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" containerName="test-curl-7325-768ecc"
Oct 30 20:27:58.632742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.632739    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" containerName="test-curl-7325-768ecc"
Oct 30 20:27:58.632957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.632905    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" containerName="test-curl-7325-768ecc"
Oct 30 20:27:58.636624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.636604    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:27:58.688076 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.688050    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-ff97ce"]
Oct 30 20:27:58.756105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.756015    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmbbg\" (UniqueName: \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\") pod \"test-curl-7325-ff97ce\" (UID: \"d69e74cb-94fb-4d8a-9470-e86223de3e08\") " pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:27:58.857786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.857750    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmbbg\" (UniqueName: \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\") pod \"test-curl-7325-ff97ce\" (UID: \"d69e74cb-94fb-4d8a-9470-e86223de3e08\") " pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:27:58.916415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.916389    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmbbg\" (UniqueName: \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\") pod \"test-curl-7325-ff97ce\" (UID: \"d69e74cb-94fb-4d8a-9470-e86223de3e08\") " pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:27:58.949427 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.949397    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:27:58.974236 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:58.973877    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-768ecc" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" containerName="test-curl-7325-768ecc" containerID="cri-o://06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74" gracePeriod=2
Oct 30 20:27:59.325181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:59.325150    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-ff97ce"]
Oct 30 20:27:59.328958 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:27:59.328928    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd69e74cb_94fb_4d8a_9470_e86223de3e08.slice/crio-c25704216e9d42f8e6070b068f292a4778cb17e8ea53c9122c06d85c001ab554 WatchSource:0}: Error finding container c25704216e9d42f8e6070b068f292a4778cb17e8ea53c9122c06d85c001ab554: Status 404 returned error can't find the container with id c25704216e9d42f8e6070b068f292a4778cb17e8ea53c9122c06d85c001ab554
Oct 30 20:27:59.950095 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:59.950065    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:27:59.984508 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:59.984475    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-ff97ce" event={"ID":"d69e74cb-94fb-4d8a-9470-e86223de3e08","Type":"ContainerStarted","Data":"046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499"}
Oct 30 20:27:59.984730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:27:59.984523    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-ff97ce" event={"ID":"d69e74cb-94fb-4d8a-9470-e86223de3e08","Type":"ContainerStarted","Data":"c25704216e9d42f8e6070b068f292a4778cb17e8ea53c9122c06d85c001ab554"}
Oct 30 20:28:01.231839 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.231765    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:28:01.286913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.286879    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\" (UniqueName: \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\") pod \"9d6c9120-2e45-490b-9f46-3418287c6079\" (UID: \"9d6c9120-2e45-490b-9f46-3418287c6079\") "
Oct 30 20:28:01.288968 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.288943    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8" pod "9d6c9120-2e45-490b-9f46-3418287c6079" (UID: "9d6c9120-2e45-490b-9f46-3418287c6079"). InnerVolumeSpecName "kube-api-access-ttxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:01.388183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.388147    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttxg8\" (UniqueName: \"kubernetes.io/projected/9d6c9120-2e45-490b-9f46-3418287c6079-kube-api-access-ttxg8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:01.718212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.718015    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-ff97ce" podStartSLOduration=3.717995462 podStartE2EDuration="3.717995462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:00.17368617 +0000 UTC m=+3789.678089920" watchObservedRunningTime="2026-10-30 20:28:01.717995462 +0000 UTC m=+3791.222399208"
Oct 30 20:28:01.718212 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.718160    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-ff97ce"]
Oct 30 20:28:01.723286 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.723254    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-ff97ce"]
Oct 30 20:28:01.866092 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.866058    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-8cfa0c"]
Oct 30 20:28:01.866735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.866720    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" containerName="test-curl-7325-ff97ce"
Oct 30 20:28:01.866791 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.866735    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" containerName="test-curl-7325-ff97ce"
Oct 30 20:28:01.866846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.866836    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" containerName="test-curl-7325-ff97ce"
Oct 30 20:28:01.870414 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.870395    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:01.924508 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.924475    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-8cfa0c"]
Oct 30 20:28:01.995478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:01.995401    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsn6\" (UniqueName: \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\") pod \"test-curl-7325-8cfa0c\" (UID: \"fa7594b7-9470-4a46-abc0-489f421d18b5\") " pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:02.002187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.002157    5085 generic.go:356] "Generic (PLEG): container finished" podID="9d6c9120-2e45-490b-9f46-3418287c6079" containerID="06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74" exitCode=137
Oct 30 20:28:02.002328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.002203    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:28:02.002328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.002250    5085 scope.go:122] "RemoveContainer" containerID="06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74"
Oct 30 20:28:02.002479 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.002451    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-ff97ce" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" containerName="test-curl-7325-ff97ce" containerID="cri-o://046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499" gracePeriod=2
Oct 30 20:28:02.012517 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.012497    5085 scope.go:122] "RemoveContainer" containerID="06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74"
Oct 30 20:28:02.012828 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:02.012809    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74\": container with ID starting with 06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74 not found: ID does not exist" containerID="06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74"
Oct 30 20:28:02.012889 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.012836    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74"} err="failed to get container status \"06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74\": rpc error: code = NotFound desc = could not find container \"06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74\": container with ID starting with 06ae7c970f0770ba0d6f7a54c06f4839f1921fb6989fd031bdbffd52a32c1b74 not found: ID does not exist"
Oct 30 20:28:02.050505 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:02.050458    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-ff97ce\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:28:02.096917 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.096878    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsn6\" (UniqueName: \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\") pod \"test-curl-7325-8cfa0c\" (UID: \"fa7594b7-9470-4a46-abc0-489f421d18b5\") " pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:02.098689 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:02.098663    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-768ecc\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" pod="opendatahub/test-curl-7325-768ecc"
Oct 30 20:28:02.146638 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:02.146597    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-ff97ce\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:28:02.156627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.156598    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsn6\" (UniqueName: \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\") pod \"test-curl-7325-8cfa0c\" (UID: \"fa7594b7-9470-4a46-abc0-489f421d18b5\") " pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:02.182811 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.182783    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:02.554659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.554624    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-8cfa0c"]
Oct 30 20:28:02.555219 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:28:02.555185    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa7594b7_9470_4a46_abc0_489f421d18b5.slice/crio-ce41aaac681453618f119a1e723cb75e201994a5b4a7be8db8c6ba905b8758b5 WatchSource:0}: Error finding container ce41aaac681453618f119a1e723cb75e201994a5b4a7be8db8c6ba905b8758b5: Status 404 returned error can't find the container with id ce41aaac681453618f119a1e723cb75e201994a5b4a7be8db8c6ba905b8758b5
Oct 30 20:28:02.633105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:02.633080    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d6c9120-2e45-490b-9f46-3418287c6079" path="/var/lib/kubelet/pods/9d6c9120-2e45-490b-9f46-3418287c6079/volumes"
Oct 30 20:28:03.011669 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:03.011633    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-8cfa0c" event={"ID":"fa7594b7-9470-4a46-abc0-489f421d18b5","Type":"ContainerStarted","Data":"6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a"}
Oct 30 20:28:03.011864 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:03.011685    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-8cfa0c" event={"ID":"fa7594b7-9470-4a46-abc0-489f421d18b5","Type":"ContainerStarted","Data":"ce41aaac681453618f119a1e723cb75e201994a5b4a7be8db8c6ba905b8758b5"}
Oct 30 20:28:03.121804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:03.121752    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-8cfa0c" podStartSLOduration=2.121737336 podStartE2EDuration="2.121737336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:03.119829477 +0000 UTC m=+3792.624233226" watchObservedRunningTime="2026-10-30 20:28:03.121737336 +0000 UTC m=+3792.626141082"
Oct 30 20:28:04.238772 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.238747    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:28:04.322891 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.322806    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\" (UniqueName: \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\") pod \"d69e74cb-94fb-4d8a-9470-e86223de3e08\" (UID: \"d69e74cb-94fb-4d8a-9470-e86223de3e08\") "
Oct 30 20:28:04.325092 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.325063    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg" pod "d69e74cb-94fb-4d8a-9470-e86223de3e08" (UID: "d69e74cb-94fb-4d8a-9470-e86223de3e08"). InnerVolumeSpecName "kube-api-access-qmbbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:04.425049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.425016    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmbbg\" (UniqueName: \"kubernetes.io/projected/d69e74cb-94fb-4d8a-9470-e86223de3e08-kube-api-access-qmbbg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:04.531326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.531293    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-8cfa0c"]
Oct 30 20:28:04.537981 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.537954    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-8cfa0c"]
Oct 30 20:28:04.632145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.632072    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69e74cb-94fb-4d8a-9470-e86223de3e08" path="/var/lib/kubelet/pods/d69e74cb-94fb-4d8a-9470-e86223de3e08/volumes"
Oct 30 20:28:04.686659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.686630    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-3e9666"]
Oct 30 20:28:04.687294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.687278    5085 cpu_manager.go:414] "containerMap: removing container" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" containerName="test-curl-7325-8cfa0c"
Oct 30 20:28:04.687294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.687293    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" containerName="test-curl-7325-8cfa0c"
Oct 30 20:28:04.687401 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.687385    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" containerName="test-curl-7325-8cfa0c"
Oct 30 20:28:04.691336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.691320    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:04.751639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.751608    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-3e9666"]
Oct 30 20:28:04.828956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.828920    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl49\" (UniqueName: \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\") pod \"test-curl-7325-3e9666\" (UID: \"1470d52d-09a4-43ca-9dfc-02403e93b6ab\") " pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:04.930478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.930445    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl49\" (UniqueName: \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\") pod \"test-curl-7325-3e9666\" (UID: \"1470d52d-09a4-43ca-9dfc-02403e93b6ab\") " pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:04.989229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:04.989193    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl49\" (UniqueName: \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\") pod \"test-curl-7325-3e9666\" (UID: \"1470d52d-09a4-43ca-9dfc-02403e93b6ab\") " pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:05.004134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.004107    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:05.028669 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.028640    5085 generic.go:356] "Generic (PLEG): container finished" podID="d69e74cb-94fb-4d8a-9470-e86223de3e08" containerID="046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499" exitCode=137
Oct 30 20:28:05.028799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.028688    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-ff97ce"
Oct 30 20:28:05.028799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.028744    5085 scope.go:122] "RemoveContainer" containerID="046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499"
Oct 30 20:28:05.029028 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.029007    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-8cfa0c" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" containerName="test-curl-7325-8cfa0c" containerID="cri-o://6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a" gracePeriod=2
Oct 30 20:28:05.043133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.043101    5085 scope.go:122] "RemoveContainer" containerID="046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499"
Oct 30 20:28:05.043412 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:05.043385    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499\": container with ID starting with 046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499 not found: ID does not exist" containerID="046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499"
Oct 30 20:28:05.043510 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.043423    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499"} err="failed to get container status \"046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499\": rpc error: code = NotFound desc = could not find container \"046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499\": container with ID starting with 046935918d75f6b0278f168c66bcf0973c2e995d01a44ac183b95ce787460499 not found: ID does not exist"
Oct 30 20:28:05.376070 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:05.376041    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-3e9666"]
Oct 30 20:28:05.377244 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:28:05.377203    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1470d52d_09a4_43ca_9dfc_02403e93b6ab.slice/crio-9b3f8c69d87e358c41f8a4c77ae6488bd4138bde3ab92573789e49de4e00c01d WatchSource:0}: Error finding container 9b3f8c69d87e358c41f8a4c77ae6488bd4138bde3ab92573789e49de4e00c01d: Status 404 returned error can't find the container with id 9b3f8c69d87e358c41f8a4c77ae6488bd4138bde3ab92573789e49de4e00c01d
Oct 30 20:28:06.039889 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:06.039850    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-3e9666" event={"ID":"1470d52d-09a4-43ca-9dfc-02403e93b6ab","Type":"ContainerStarted","Data":"88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298"}
Oct 30 20:28:06.039889 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:06.039897    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-3e9666" event={"ID":"1470d52d-09a4-43ca-9dfc-02403e93b6ab","Type":"ContainerStarted","Data":"9b3f8c69d87e358c41f8a4c77ae6488bd4138bde3ab92573789e49de4e00c01d"}
Oct 30 20:28:06.151182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:06.151127    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-3e9666" podStartSLOduration=2.151112642 podStartE2EDuration="2.151112642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:06.14778597 +0000 UTC m=+3795.652189717" watchObservedRunningTime="2026-10-30 20:28:06.151112642 +0000 UTC m=+3795.655516389"
Oct 30 20:28:06.162418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:06.162384    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:28:07.317455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.317430    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:07.460594 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.460545    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\" (UniqueName: \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\") pod \"fa7594b7-9470-4a46-abc0-489f421d18b5\" (UID: \"fa7594b7-9470-4a46-abc0-489f421d18b5\") "
Oct 30 20:28:07.462670 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.462633    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6" pod "fa7594b7-9470-4a46-abc0-489f421d18b5" (UID: "fa7594b7-9470-4a46-abc0-489f421d18b5"). InnerVolumeSpecName "kube-api-access-5bsn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:07.562506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.562422    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bsn6\" (UniqueName: \"kubernetes.io/projected/fa7594b7-9470-4a46-abc0-489f421d18b5-kube-api-access-5bsn6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:07.655770 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.655729    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-3e9666"]
Oct 30 20:28:07.660463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:07.660426    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-3e9666"]
Oct 30 20:28:08.056890 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.056851    5085 generic.go:356] "Generic (PLEG): container finished" podID="fa7594b7-9470-4a46-abc0-489f421d18b5" containerID="6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a" exitCode=137
Oct 30 20:28:08.057075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.056899    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:08.057075 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.056944    5085 scope.go:122] "RemoveContainer" containerID="6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a"
Oct 30 20:28:08.057225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.057146    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-3e9666" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" containerName="test-curl-7325-3e9666" containerID="cri-o://88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298" gracePeriod=2
Oct 30 20:28:08.071044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.071018    5085 scope.go:122] "RemoveContainer" containerID="6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a"
Oct 30 20:28:08.071388 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:08.071366    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a\": container with ID starting with 6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a not found: ID does not exist" containerID="6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a"
Oct 30 20:28:08.071478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.071398    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a"} err="failed to get container status \"6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a\": rpc error: code = NotFound desc = could not find container \"6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a\": container with ID starting with 6dbcb757a04db8efb66c95696e4c474c2ebf76e6e66259d299dafdab468ca41a not found: ID does not exist"
Oct 30 20:28:08.105277 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:08.105239    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-3e9666\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:08.154017 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:08.153977    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-8cfa0c\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" pod="opendatahub/test-curl-7325-8cfa0c"
Oct 30 20:28:08.202110 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:08.202069    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-3e9666\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:08.631736 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:08.631693    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7594b7-9470-4a46-abc0-489f421d18b5" path="/var/lib/kubelet/pods/fa7594b7-9470-4a46-abc0-489f421d18b5/volumes"
Oct 30 20:28:10.302747 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:10.302720    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:10.400975 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:10.400903    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\" (UniqueName: \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\") pod \"1470d52d-09a4-43ca-9dfc-02403e93b6ab\" (UID: \"1470d52d-09a4-43ca-9dfc-02403e93b6ab\") "
Oct 30 20:28:10.402921 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:10.402891    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49" pod "1470d52d-09a4-43ca-9dfc-02403e93b6ab" (UID: "1470d52d-09a4-43ca-9dfc-02403e93b6ab"). InnerVolumeSpecName "kube-api-access-4pl49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:10.502584 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:10.502527    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl49\" (UniqueName: \"kubernetes.io/projected/1470d52d-09a4-43ca-9dfc-02403e93b6ab-kube-api-access-4pl49\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:10.632368 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:10.632336    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" path="/var/lib/kubelet/pods/1470d52d-09a4-43ca-9dfc-02403e93b6ab/volumes"
Oct 30 20:28:11.079831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:11.079796    5085 generic.go:356] "Generic (PLEG): container finished" podID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" containerID="88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298" exitCode=137
Oct 30 20:28:11.080036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:11.079839    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-3e9666"
Oct 30 20:28:11.080036 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:11.079887    5085 scope.go:122] "RemoveContainer" containerID="88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298"
Oct 30 20:28:11.089708 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:11.089690    5085 scope.go:122] "RemoveContainer" containerID="88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298"
Oct 30 20:28:11.089968 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:11.089951    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298\": container with ID starting with 88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298 not found: ID does not exist" containerID="88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298"
Oct 30 20:28:11.090050 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:11.089978    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298"} err="failed to get container status \"88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298\": rpc error: code = NotFound desc = could not find container \"88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298\": container with ID starting with 88142d9f87e5dae9b843fba05eeb1eed08297fe5681a7296d6d81abd633d7298 not found: ID does not exist"
Oct 30 20:28:21.940757 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:21.940714    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:28:45.096056 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:45.096005    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:28:46.395614 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.395555    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7334-2f919b"]
Oct 30 20:28:46.396261 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.396245    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" containerName="test-curl-7325-3e9666"
Oct 30 20:28:46.396325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.396263    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" containerName="test-curl-7325-3e9666"
Oct 30 20:28:46.396364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.396354    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1470d52d-09a4-43ca-9dfc-02403e93b6ab" containerName="test-curl-7325-3e9666"
Oct 30 20:28:46.400257 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.400237    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:46.453421 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.453385    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-2f919b"]
Oct 30 20:28:46.502249 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.500763    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\""
Oct 30 20:28:46.525146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.525110    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz6tj\" (UniqueName: \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\") pod \"test-curl-7334-2f919b\" (UID: \"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9\") " pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:46.626168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.626130    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz6tj\" (UniqueName: \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\") pod \"test-curl-7334-2f919b\" (UID: \"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9\") " pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:46.702158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.702122    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz6tj\" (UniqueName: \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\") pod \"test-curl-7334-2f919b\" (UID: \"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9\") " pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:46.723691 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:46.723661    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:47.120348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:47.120319    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-2f919b"]
Oct 30 20:28:47.122308 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:28:47.121350    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c21a8a_10e9_4bc5_8e2b_3225445ee8f9.slice/crio-7dc40d25dd8057de50528acd0ff4043ed8d94e47b70e86e731c55edb2c974df9 WatchSource:0}: Error finding container 7dc40d25dd8057de50528acd0ff4043ed8d94e47b70e86e731c55edb2c974df9: Status 404 returned error can't find the container with id 7dc40d25dd8057de50528acd0ff4043ed8d94e47b70e86e731c55edb2c974df9
Oct 30 20:28:47.366051 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:47.366017    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-2f919b" event={"ID":"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9","Type":"ContainerStarted","Data":"092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5"}
Oct 30 20:28:47.366244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:47.366062    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-2f919b" event={"ID":"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9","Type":"ContainerStarted","Data":"7dc40d25dd8057de50528acd0ff4043ed8d94e47b70e86e731c55edb2c974df9"}
Oct 30 20:28:47.475064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:47.475006    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7334-2f919b" podStartSLOduration=1.474990469 podStartE2EDuration="1.474990469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:47.473759187 +0000 UTC m=+3836.978162935" watchObservedRunningTime="2026-10-30 20:28:47.474990469 +0000 UTC m=+3836.979394216"
Oct 30 20:28:48.922060 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:48.922026    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7334-2f919b"]
Oct 30 20:28:48.927172 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:48.927139    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7334-2f919b"]
Oct 30 20:28:49.380327 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:49.380231    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7334-2f919b" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" containerName="test-curl-7334-2f919b" containerID="cri-o://092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5" gracePeriod=2
Oct 30 20:28:49.428436 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:49.428398    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-2f919b\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:50.676062 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:50.676021    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-2f919b\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:51.639360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:51.639303    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:51.680631 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:51.680597    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\" (UniqueName: \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\") pod \"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9\" (UID: \"27c21a8a-10e9-4bc5-8e2b-3225445ee8f9\") "
Oct 30 20:28:51.682804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:51.682778    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj" pod "27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" (UID: "27c21a8a-10e9-4bc5-8e2b-3225445ee8f9"). InnerVolumeSpecName "kube-api-access-wz6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:51.782191 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:51.782151    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz6tj\" (UniqueName: \"kubernetes.io/projected/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9-kube-api-access-wz6tj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:28:51.940731 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:51.940696    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:28:52.409977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.409893    5085 generic.go:356] "Generic (PLEG): container finished" podID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" containerID="092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5" exitCode=137
Oct 30 20:28:52.409977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.409930    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:52.410218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.409980    5085 scope.go:122] "RemoveContainer" containerID="092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5"
Oct 30 20:28:52.419935 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.419917    5085 scope.go:122] "RemoveContainer" containerID="092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5"
Oct 30 20:28:52.420200 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:52.420181    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5\": container with ID starting with 092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5 not found: ID does not exist" containerID="092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5"
Oct 30 20:28:52.420253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.420208    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5"} err="failed to get container status \"092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5\": rpc error: code = NotFound desc = could not find container \"092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5\": container with ID starting with 092d1a52b6490f6a18df70c62a1a26b982b94c09e1cad20819675ba529c355f5 not found: ID does not exist"
Oct 30 20:28:52.466551 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:52.466516    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-2f919b\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" pod="opendatahub/test-curl-7334-2f919b"
Oct 30 20:28:52.633421 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:52.633389    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" path="/var/lib/kubelet/pods/27c21a8a-10e9-4bc5-8e2b-3225445ee8f9/volumes"
Oct 30 20:28:53.935416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.935380    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7331-9798b1"]
Oct 30 20:28:53.936292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.936269    5085 cpu_manager.go:414] "containerMap: removing container" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" containerName="test-curl-7334-2f919b"
Oct 30 20:28:53.936292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.936290    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" containerName="test-curl-7334-2f919b"
Oct 30 20:28:53.936460 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.936440    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c21a8a-10e9-4bc5-8e2b-3225445ee8f9" containerName="test-curl-7334-2f919b"
Oct 30 20:28:53.941295 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.941272    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:53.990408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:53.990372    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7331-9798b1"]
Oct 30 20:28:54.005516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.005484    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxbjv\" (UniqueName: \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\") pod \"test-curl-7331-9798b1\" (UID: \"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a\") " pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:54.032590 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.032536    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-f4jr9\""
Oct 30 20:28:54.107169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.107131    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbjv\" (UniqueName: \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\") pod \"test-curl-7331-9798b1\" (UID: \"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a\") " pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:54.165598 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.165528    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbjv\" (UniqueName: \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\") pod \"test-curl-7331-9798b1\" (UID: \"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a\") " pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:54.253884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.253813    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:54.626664 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:28:54.626634    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9b65f8c_378c_48c9_a6f2_d38b2c2c3e4a.slice/crio-6470d35f28491697ad631253da89dca2cde0080dc8ed9339dbd21842b4314394 WatchSource:0}: Error finding container 6470d35f28491697ad631253da89dca2cde0080dc8ed9339dbd21842b4314394: Status 404 returned error can't find the container with id 6470d35f28491697ad631253da89dca2cde0080dc8ed9339dbd21842b4314394
Oct 30 20:28:54.633276 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:54.633250    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7331-9798b1"]
Oct 30 20:28:55.436634 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:55.436598    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7331-9798b1" event={"ID":"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a","Type":"ContainerStarted","Data":"7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a"}
Oct 30 20:28:55.437130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:55.436645    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7331-9798b1" event={"ID":"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a","Type":"ContainerStarted","Data":"6470d35f28491697ad631253da89dca2cde0080dc8ed9339dbd21842b4314394"}
Oct 30 20:28:55.544327 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:55.544276    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7331-9798b1" podStartSLOduration=2.544261874 podStartE2EDuration="2.544261874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:55.543695122 +0000 UTC m=+3845.048098869" watchObservedRunningTime="2026-10-30 20:28:55.544261874 +0000 UTC m=+3845.048665621"
Oct 30 20:28:57.047623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:57.047587    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7331-9798b1"]
Oct 30 20:28:57.052624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:57.052597    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7331-9798b1"]
Oct 30 20:28:57.451748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:57.451696    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7331-9798b1" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" containerName="test-curl-7331-9798b1" containerID="cri-o://7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a" gracePeriod=2
Oct 30 20:28:57.500140 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:28:57.500100    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7331-9798b1\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:59.704843 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:59.704814    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:28:59.778029 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:59.777947    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\" (UniqueName: \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\") pod \"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a\" (UID: \"d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a\") "
Oct 30 20:28:59.780226 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:59.780198    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv" pod "d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" (UID: "d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a"). InnerVolumeSpecName "kube-api-access-xxbjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:28:59.880094 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:28:59.880050    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxbjv\" (UniqueName: \"kubernetes.io/projected/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a-kube-api-access-xxbjv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:29:00.507180 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.507146    5085 generic.go:356] "Generic (PLEG): container finished" podID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" containerID="7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a" exitCode=137
Oct 30 20:29:00.507360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.507202    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:29:00.507360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.507248    5085 scope.go:122] "RemoveContainer" containerID="7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a"
Oct 30 20:29:00.517044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.517029    5085 scope.go:122] "RemoveContainer" containerID="7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a"
Oct 30 20:29:00.517278 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:00.517262    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a\": container with ID starting with 7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a not found: ID does not exist" containerID="7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a"
Oct 30 20:29:00.517334 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.517284    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a"} err="failed to get container status \"7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a\": rpc error: code = NotFound desc = could not find container \"7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a\": container with ID starting with 7785c9167b3bb9d65c7bfdbbb709eff40cbce4e293a1be7d9aefa3bf69c4e24a not found: ID does not exist"
Oct 30 20:29:00.569000 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:00.568970    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7331-9798b1\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:29:00.632110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:00.632082    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" path="/var/lib/kubelet/pods/d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a/volumes"
Oct 30 20:29:00.675897 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:00.675857    5085 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7331-9798b1\" is forbidden: User \"system:node:ip-10-0-134-200.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-200.us-west-2.compute.internal' and this object" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" pod="opendatahub/test-curl-7331-9798b1"
Oct 30 20:29:21.949513 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:21.949472    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:29:25.704732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.704703    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:29:25.705384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.705369    5085 cpu_manager.go:414] "containerMap: removing container" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" containerName="test-curl-7331-9798b1"
Oct 30 20:29:25.705435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.705385    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" containerName="test-curl-7331-9798b1"
Oct 30 20:29:25.705499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.705490    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b65f8c-378c-48c9-a6f2-d38b2c2c3e4a" containerName="test-curl-7331-9798b1"
Oct 30 20:29:25.709275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.709255    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.759110 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.759076    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0e72bd-serving-cert\""
Oct 30 20:29:25.809213 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.809184    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:29:25.810362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.810340    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.810515 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.810492    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.810595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.810579    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.810647 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.810629    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4596v\" (UniqueName: \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.912145 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.912105    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.912336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.912172    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.912336 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.912201    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4596v\" (UniqueName: \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.912336 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:25.912284    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-serving-cert: secret "maas-api-e2e-extmod-a-0e72bd-serving-cert" not found
Oct 30 20:29:25.912488 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.912294    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.912488 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:25.912361    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls podName:af006b12-d6df-4191-b47a-7f3100f1416b nodeName:}" failed. No retries permitted until 2026-10-30 20:29:26.412338072 +0000 UTC m=+3875.916741839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls") pod "maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" (UID: "af006b12-d6df-4191-b47a-7f3100f1416b") : secret "maas-api-e2e-extmod-a-0e72bd-serving-cert" not found
Oct 30 20:29:25.912898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.912869    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.914755 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.914733    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:25.971248 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:25.971174    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4596v\" (UniqueName: \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:26.419419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:26.419307    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:26.421816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:26.421794    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:26.622474 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:26.622446    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:26.996372 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:26.996342    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:29:26.997613 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:29:26.997583    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf006b12_d6df_4191_b47a_7f3100f1416b.slice/crio-065a218983bf73f4c0b32dd93c05d22acd0bedb6d4b96422728fcc2d8ad055f6 WatchSource:0}: Error finding container 065a218983bf73f4c0b32dd93c05d22acd0bedb6d4b96422728fcc2d8ad055f6: Status 404 returned error can't find the container with id 065a218983bf73f4c0b32dd93c05d22acd0bedb6d4b96422728fcc2d8ad055f6
Oct 30 20:29:27.705100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:27.705062    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" event={"ID":"af006b12-d6df-4191-b47a-7f3100f1416b","Type":"ContainerStarted","Data":"065a218983bf73f4c0b32dd93c05d22acd0bedb6d4b96422728fcc2d8ad055f6"}
Oct 30 20:29:28.714981 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:28.714947    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" event={"ID":"af006b12-d6df-4191-b47a-7f3100f1416b","Type":"ContainerStarted","Data":"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"}
Oct 30 20:29:28.715375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:28.715024    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:28.830624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:28.830548    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" podStartSLOduration=2.996062978 podStartE2EDuration="3.830534215s" totalImagesPullingTime="834.471237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:28.828950873 +0000 UTC m=+3878.333354620" watchObservedRunningTime="2026-10-30 20:29:28.830534215 +0000 UTC m=+3878.334937963"
Oct 30 20:29:30.044867 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:30.044829    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:29:34.727487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:34.727457    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:29:51.538924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:51.538881    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:29:51.550208 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:29:51.550182    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:29:51.940098 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:29:51.940058    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:30:13.957192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:13.957154    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:30:13.962924 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:13.962902    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.048595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.048522    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.048595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.048597    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.048856 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.048683    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxzj\" (UniqueName: \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.048856 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.048742    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.060589 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.060548    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9389b9-serving-cert\""
Oct 30 20:30:14.064199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.064177    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:30:14.150015 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.149975    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.150209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.150040    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.150209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.150147    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxzj\" (UniqueName: \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.150310 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:14.150207    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-serving-cert: secret "maas-api-e2e-extmod-a-9389b9-serving-cert" not found
Oct 30 20:30:14.150310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.150221    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.150310 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:14.150280    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls podName:5098658d-9bc1-41cd-a002-e60fd349937c nodeName:}" failed. No retries permitted until 2026-10-30 20:30:14.650257416 +0000 UTC m=+3924.154661153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls") pod "maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" (UID: "5098658d-9bc1-41cd-a002-e60fd349937c") : secret "maas-api-e2e-extmod-a-9389b9-serving-cert" not found
Oct 30 20:30:14.150725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.150704    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.152711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.152692    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.209032 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.208970    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxzj\" (UniqueName: \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.656557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.656484    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.658941 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.658917    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:14.877253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:14.877208    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:15.280943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:15.280903    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:30:16.046170 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:16.046132    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" event={"ID":"5098658d-9bc1-41cd-a002-e60fd349937c","Type":"ContainerStarted","Data":"3bb2614f5425205e2c3228c92b2e452c68a0a6678e9bdde0aa87dd0501dfe455"}
Oct 30 20:30:17.055458 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:17.055408    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" event={"ID":"5098658d-9bc1-41cd-a002-e60fd349937c","Type":"ContainerStarted","Data":"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"}
Oct 30 20:30:17.055881 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:17.055487    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:17.174809 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:17.174757    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" podStartSLOduration=3.213455915 podStartE2EDuration="4.174741782s" totalImagesPullingTime="961.285867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:17.172169265 +0000 UTC m=+3926.676573011" watchObservedRunningTime="2026-10-30 20:30:17.174741782 +0000 UTC m=+3926.679145529"
Oct 30 20:30:21.940397 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:21.940355    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:30:22.612872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:22.612834    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:30:23.068545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:23.068516    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:30:29.936737 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:29.936701    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:30:29.941178 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:29.941163    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.039548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.039512    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8d612c-serving-cert\""
Oct 30 20:30:30.040171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.040144    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bhns\" (UniqueName: \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.040242 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.040209    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.040325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.040308    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.040494 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.040472    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.043416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.043396    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:30:30.142179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.142144    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.142384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.142213    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bhns\" (UniqueName: \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.142384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.142276    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.142384 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.142347    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.142609 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:30.142446    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-serving-cert: secret "maas-api-e2e-extmod-a-8d612c-serving-cert" not found
Oct 30 20:30:30.142609 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:30.142520    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls podName:2921d3c0-917b-4902-9263-5b667829adfc nodeName:}" failed. No retries permitted until 2026-10-30 20:30:30.64249753 +0000 UTC m=+3940.146901274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls") pod "maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" (UID: "2921d3c0-917b-4902-9263-5b667829adfc") : secret "maas-api-e2e-extmod-a-8d612c-serving-cert" not found
Oct 30 20:30:30.142954 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.142931    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.145035 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.145012    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.206286 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.206216    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bhns\" (UniqueName: \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.648661 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.648575    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.651407 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.651380    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:30.856652 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:30.856613    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:31.232960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:31.232924    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:30:31.233344 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:30:31.233159    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2921d3c0_917b_4902_9263_5b667829adfc.slice/crio-215a28c4e0cbb2078973cbb15b092e82c545a3de4481696dc2291f6bbe2a4c36 WatchSource:0}: Error finding container 215a28c4e0cbb2078973cbb15b092e82c545a3de4481696dc2291f6bbe2a4c36: Status 404 returned error can't find the container with id 215a28c4e0cbb2078973cbb15b092e82c545a3de4481696dc2291f6bbe2a4c36
Oct 30 20:30:32.167260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:32.167232    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" event={"ID":"2921d3c0-917b-4902-9263-5b667829adfc","Type":"ContainerStarted","Data":"215a28c4e0cbb2078973cbb15b092e82c545a3de4481696dc2291f6bbe2a4c36"}
Oct 30 20:30:33.179535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:33.179500    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" event={"ID":"2921d3c0-917b-4902-9263-5b667829adfc","Type":"ContainerStarted","Data":"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"}
Oct 30 20:30:33.179950 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:33.179604    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:33.293038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:33.292987    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" podStartSLOduration=3.384048686 podStartE2EDuration="4.292972335s" totalImagesPullingTime="908.923649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:33.290835855 +0000 UTC m=+3942.795239601" watchObservedRunningTime="2026-10-30 20:30:33.292972335 +0000 UTC m=+3942.797376082"
Oct 30 20:30:39.191878 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:30:39.191846    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:30:51.939937 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:30:51.939903    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:31:21.940146 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:31:21.940108    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:31:33.446313 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:33.446141    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:31:51.940201 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:31:51.940159    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:31:52.234163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.234037    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:31:52.238788 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.238769    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.290476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.290440    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-79c649-serving-cert\""
Oct 30 20:31:52.295023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.294999    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:31:52.320557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.320526    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.320723 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.320658    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.320806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.320784    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.320901 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.320883    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptvf\" (UniqueName: \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.422177 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.422147    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptvf\" (UniqueName: \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.422367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.422265    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.422367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.422301    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.422501 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.422467    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.422610 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:31:52.422589    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-79c649-serving-cert: secret "maas-api-e2e-autores-79c649-serving-cert" not found
Oct 30 20:31:52.422688 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:31:52.422662    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls podName:bfec0531-34f8-4963-95f6-18e1d1d757e9 nodeName:}" failed. No retries permitted until 2026-10-30 20:31:52.922642572 +0000 UTC m=+4022.427046300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls") pod "maas-api-e2e-autores-79c649-86bd595875-bf5qm" (UID: "bfec0531-34f8-4963-95f6-18e1d1d757e9") : secret "maas-api-e2e-autores-79c649-serving-cert" not found
Oct 30 20:31:52.423115 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.423093    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.424831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.424809    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.482620 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.482582    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptvf\" (UniqueName: \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.929416 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.929383    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:52.931843 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:52.931823    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") pod \"maas-api-e2e-autores-79c649-86bd595875-bf5qm\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:53.152622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:53.152549    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:53.527713 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:53.527682    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:31:53.529144 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:31:53.529115    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfec0531_34f8_4963_95f6_18e1d1d757e9.slice/crio-0ad06fe1435ef0ef23a8470b88565d40c13c5096c25c0c372ba1f121706d19a4 WatchSource:0}: Error finding container 0ad06fe1435ef0ef23a8470b88565d40c13c5096c25c0c372ba1f121706d19a4: Status 404 returned error can't find the container with id 0ad06fe1435ef0ef23a8470b88565d40c13c5096c25c0c372ba1f121706d19a4
Oct 30 20:31:53.740263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:53.740223    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" event={"ID":"bfec0531-34f8-4963-95f6-18e1d1d757e9","Type":"ContainerStarted","Data":"0ad06fe1435ef0ef23a8470b88565d40c13c5096c25c0c372ba1f121706d19a4"}
Oct 30 20:31:54.749229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:54.749197    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" event={"ID":"bfec0531-34f8-4963-95f6-18e1d1d757e9","Type":"ContainerStarted","Data":"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"}
Oct 30 20:31:54.749643 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:54.749412    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:31:54.867434 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:31:54.867330    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" podStartSLOduration=1.872294108 podStartE2EDuration="2.867315991s" totalImagesPullingTime="995.021883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:54.863545349 +0000 UTC m=+4024.367949096" watchObservedRunningTime="2026-10-30 20:31:54.867315991 +0000 UTC m=+4024.371719760"
Oct 30 20:32:00.763067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:32:00.763037    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:32:13.807698 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:32:13.807655    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:32:21.940160 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:32:21.940123    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:32:51.939781 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:32:51.939737    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:33:21.694487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:21.694448    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:33:21.943650 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:21.943613    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:33:39.872703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:39.872669    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:33:39.873189 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:39.872928    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" podUID="bfec0531-34f8-4963-95f6-18e1d1d757e9" containerName="maas-api" containerID="cri-o://f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f" gracePeriod=30
Oct 30 20:33:40.140734 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.140671    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:33:40.218492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.218457    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\") pod \"bfec0531-34f8-4963-95f6-18e1d1d757e9\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") "
Oct 30 20:33:40.218695 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.218523    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\" (UniqueName: \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\") pod \"bfec0531-34f8-4963-95f6-18e1d1d757e9\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") "
Oct 30 20:33:40.218695 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.218655    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\") pod \"bfec0531-34f8-4963-95f6-18e1d1d757e9\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") "
Oct 30 20:33:40.218695 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.218685    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") pod \"bfec0531-34f8-4963-95f6-18e1d1d757e9\" (UID: \"bfec0531-34f8-4963-95f6-18e1d1d757e9\") "
Oct 30 20:33:40.218895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.218819    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca" pod "bfec0531-34f8-4963-95f6-18e1d1d757e9" (UID: "bfec0531-34f8-4963-95f6-18e1d1d757e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:40.219100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.219080    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfec0531-34f8-4963-95f6-18e1d1d757e9-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:40.220781 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.220756    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs" pod "bfec0531-34f8-4963-95f6-18e1d1d757e9" (UID: "bfec0531-34f8-4963-95f6-18e1d1d757e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:40.220781 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.220766    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls" pod "bfec0531-34f8-4963-95f6-18e1d1d757e9" (UID: "bfec0531-34f8-4963-95f6-18e1d1d757e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:40.220932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.220769    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf" pod "bfec0531-34f8-4963-95f6-18e1d1d757e9" (UID: "bfec0531-34f8-4963-95f6-18e1d1d757e9"). InnerVolumeSpecName "kube-api-access-mptvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:40.320535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.320499    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:40.320535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.320535    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfec0531-34f8-4963-95f6-18e1d1d757e9-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:40.320764 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.320551    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptvf\" (UniqueName: \"kubernetes.io/projected/bfec0531-34f8-4963-95f6-18e1d1d757e9-kube-api-access-mptvf\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:40.554761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.554728    5085 generic.go:356] "Generic (PLEG): container finished" podID="bfec0531-34f8-4963-95f6-18e1d1d757e9" containerID="f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f" exitCode=0
Oct 30 20:33:40.554986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.554778    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"
Oct 30 20:33:40.554986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.554805    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" event={"ID":"bfec0531-34f8-4963-95f6-18e1d1d757e9","Type":"ContainerDied","Data":"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"}
Oct 30 20:33:40.554986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.554863    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm" event={"ID":"bfec0531-34f8-4963-95f6-18e1d1d757e9","Type":"ContainerDied","Data":"0ad06fe1435ef0ef23a8470b88565d40c13c5096c25c0c372ba1f121706d19a4"}
Oct 30 20:33:40.554986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.554885    5085 scope.go:122] "RemoveContainer" containerID="f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"
Oct 30 20:33:40.567058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.567038    5085 scope.go:122] "RemoveContainer" containerID="f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"
Oct 30 20:33:40.567312 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:40.567296    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f\": container with ID starting with f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f not found: ID does not exist" containerID="f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"
Oct 30 20:33:40.567361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.567319    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f"} err="failed to get container status \"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f\": rpc error: code = NotFound desc = could not find container \"f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f\": container with ID starting with f6b3e44f635bee023d5150564d188c865f926802daa0d6587e7dad6e5cd6186f not found: ID does not exist"
Oct 30 20:33:40.714064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.714033    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:33:40.720300 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:40.720270    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-79c649-86bd595875-bf5qm"]
Oct 30 20:33:41.072020 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.071983    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:33:41.072478 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.072253    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" podUID="5098658d-9bc1-41cd-a002-e60fd349937c" containerName="maas-api" containerID="cri-o://68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d" gracePeriod=30
Oct 30 20:33:41.364515 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.364486    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:33:41.435070 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435035    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\") pod \"5098658d-9bc1-41cd-a002-e60fd349937c\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") "
Oct 30 20:33:41.435262 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435104    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") pod \"5098658d-9bc1-41cd-a002-e60fd349937c\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") "
Oct 30 20:33:41.435324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435300    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\") pod \"5098658d-9bc1-41cd-a002-e60fd349937c\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") "
Oct 30 20:33:41.435385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435336    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\" (UniqueName: \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\") pod \"5098658d-9bc1-41cd-a002-e60fd349937c\" (UID: \"5098658d-9bc1-41cd-a002-e60fd349937c\") "
Oct 30 20:33:41.435385 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435356    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca" pod "5098658d-9bc1-41cd-a002-e60fd349937c" (UID: "5098658d-9bc1-41cd-a002-e60fd349937c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:41.435775 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.435757    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5098658d-9bc1-41cd-a002-e60fd349937c-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.437278 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.437253    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs" pod "5098658d-9bc1-41cd-a002-e60fd349937c" (UID: "5098658d-9bc1-41cd-a002-e60fd349937c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:41.437380 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.437354    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls" pod "5098658d-9bc1-41cd-a002-e60fd349937c" (UID: "5098658d-9bc1-41cd-a002-e60fd349937c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:41.437703 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.437689    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj" pod "5098658d-9bc1-41cd-a002-e60fd349937c" (UID: "5098658d-9bc1-41cd-a002-e60fd349937c"). InnerVolumeSpecName "kube-api-access-jcxzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:41.536496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.536464    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.536496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.536492    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxzj\" (UniqueName: \"kubernetes.io/projected/5098658d-9bc1-41cd-a002-e60fd349937c-kube-api-access-jcxzj\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.536496 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.536503    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5098658d-9bc1-41cd-a002-e60fd349937c-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:41.563378 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.563346    5085 generic.go:356] "Generic (PLEG): container finished" podID="5098658d-9bc1-41cd-a002-e60fd349937c" containerID="68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d" exitCode=0
Oct 30 20:33:41.563535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.563397    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"
Oct 30 20:33:41.563535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.563426    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" event={"ID":"5098658d-9bc1-41cd-a002-e60fd349937c","Type":"ContainerDied","Data":"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"}
Oct 30 20:33:41.563535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.563479    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx" event={"ID":"5098658d-9bc1-41cd-a002-e60fd349937c","Type":"ContainerDied","Data":"3bb2614f5425205e2c3228c92b2e452c68a0a6678e9bdde0aa87dd0501dfe455"}
Oct 30 20:33:41.563535 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.563501    5085 scope.go:122] "RemoveContainer" containerID="68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"
Oct 30 20:33:41.576117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.576090    5085 scope.go:122] "RemoveContainer" containerID="68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"
Oct 30 20:33:41.576414 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:41.576391    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d\": container with ID starting with 68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d not found: ID does not exist" containerID="68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"
Oct 30 20:33:41.576477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.576422    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d"} err="failed to get container status \"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d\": rpc error: code = NotFound desc = could not find container \"68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d\": container with ID starting with 68c66d8fd1800afbbe4291366875c39ce349af2ca57e8b4d2e3de5353275d93d not found: ID does not exist"
Oct 30 20:33:41.725615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.725553    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:33:41.732233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:41.732204    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9389b9-fcfc64f5f-76smx"]
Oct 30 20:33:42.632371 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:42.632339    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5098658d-9bc1-41cd-a002-e60fd349937c" path="/var/lib/kubelet/pods/5098658d-9bc1-41cd-a002-e60fd349937c/volumes"
Oct 30 20:33:42.632867 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:42.632850    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfec0531-34f8-4963-95f6-18e1d1d757e9" path="/var/lib/kubelet/pods/bfec0531-34f8-4963-95f6-18e1d1d757e9/volumes"
Oct 30 20:33:47.591228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:47.591192    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:33:47.591882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:47.591853    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" podUID="1212b686-9e45-4df9-832a-859013d89286" containerName="maas-api" containerID="cri-o://6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f" gracePeriod=30
Oct 30 20:33:47.874796 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:47.874771    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:33:48.015081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.015028    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\") pod \"1212b686-9e45-4df9-832a-859013d89286\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") "
Oct 30 20:33:48.015282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.015132    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\") pod \"1212b686-9e45-4df9-832a-859013d89286\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") "
Oct 30 20:33:48.015282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.015245    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\" (UniqueName: \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\") pod \"1212b686-9e45-4df9-832a-859013d89286\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") "
Oct 30 20:33:48.015418 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.015299    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") pod \"1212b686-9e45-4df9-832a-859013d89286\" (UID: \"1212b686-9e45-4df9-832a-859013d89286\") "
Oct 30 20:33:48.017586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.017521    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca" pod "1212b686-9e45-4df9-832a-859013d89286" (UID: "1212b686-9e45-4df9-832a-859013d89286"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:48.020704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.020671    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs" pod "1212b686-9e45-4df9-832a-859013d89286" (UID: "1212b686-9e45-4df9-832a-859013d89286"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:48.020704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.020693    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls" pod "1212b686-9e45-4df9-832a-859013d89286" (UID: "1212b686-9e45-4df9-832a-859013d89286"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:48.020704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.020706    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg" pod "1212b686-9e45-4df9-832a-859013d89286" (UID: "1212b686-9e45-4df9-832a-859013d89286"). InnerVolumeSpecName "kube-api-access-97pgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:48.116755 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.116618    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:48.116755 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.116652    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1212b686-9e45-4df9-832a-859013d89286-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:48.116755 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.116667    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1212b686-9e45-4df9-832a-859013d89286-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:48.116755 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.116681    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97pgg\" (UniqueName: \"kubernetes.io/projected/1212b686-9e45-4df9-832a-859013d89286-kube-api-access-97pgg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:48.621842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.621806    5085 generic.go:356] "Generic (PLEG): container finished" podID="1212b686-9e45-4df9-832a-859013d89286" containerID="6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f" exitCode=0
Oct 30 20:33:48.622362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.621864    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"
Oct 30 20:33:48.622362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.621900    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" event={"ID":"1212b686-9e45-4df9-832a-859013d89286","Type":"ContainerDied","Data":"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"}
Oct 30 20:33:48.622362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.621974    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl" event={"ID":"1212b686-9e45-4df9-832a-859013d89286","Type":"ContainerDied","Data":"28dd2f19b30b6c2ff621bc5ed3984e024e5c19d1745a04930b1ca8d6660a116e"}
Oct 30 20:33:48.622362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.622007    5085 scope.go:122] "RemoveContainer" containerID="6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"
Oct 30 20:33:48.649434 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.649401    5085 scope.go:122] "RemoveContainer" containerID="6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"
Oct 30 20:33:48.649768 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:48.649745    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f\": container with ID starting with 6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f not found: ID does not exist" containerID="6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"
Oct 30 20:33:48.649884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.649775    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f"} err="failed to get container status \"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f\": rpc error: code = NotFound desc = could not find container \"6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f\": container with ID starting with 6ec1f37d0768e87cc2ec7f823bf02ad6cde194847003b4700c4a20717b2cf18f not found: ID does not exist"
Oct 30 20:33:48.780632 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.780600    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:33:48.827192 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.827155    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-742ff8-64f555cdc4-5z2wl"]
Oct 30 20:33:48.830393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.830365    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:33:48.830628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:48.830582    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" podUID="2921d3c0-917b-4902-9263-5b667829adfc" containerName="maas-api" containerID="cri-o://d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811" gracePeriod=30
Oct 30 20:33:49.107555 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.107527    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:33:49.231816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.231726    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\") pod \"2921d3c0-917b-4902-9263-5b667829adfc\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") "
Oct 30 20:33:49.231999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.231841    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\") pod \"2921d3c0-917b-4902-9263-5b667829adfc\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") "
Oct 30 20:33:49.231999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.231896    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\" (UniqueName: \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\") pod \"2921d3c0-917b-4902-9263-5b667829adfc\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") "
Oct 30 20:33:49.231999 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.231936    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") pod \"2921d3c0-917b-4902-9263-5b667829adfc\" (UID: \"2921d3c0-917b-4902-9263-5b667829adfc\") "
Oct 30 20:33:49.232195 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.232153    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca" pod "2921d3c0-917b-4902-9263-5b667829adfc" (UID: "2921d3c0-917b-4902-9263-5b667829adfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:33:49.232607 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.232551    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2921d3c0-917b-4902-9263-5b667829adfc-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.234706 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.234669    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls" pod "2921d3c0-917b-4902-9263-5b667829adfc" (UID: "2921d3c0-917b-4902-9263-5b667829adfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:49.234962 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.234939    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns" pod "2921d3c0-917b-4902-9263-5b667829adfc" (UID: "2921d3c0-917b-4902-9263-5b667829adfc"). InnerVolumeSpecName "kube-api-access-6bhns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:33:49.235083 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.235005    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs" pod "2921d3c0-917b-4902-9263-5b667829adfc" (UID: "2921d3c0-917b-4902-9263-5b667829adfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:33:49.334223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.334191    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.334223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.334220    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bhns\" (UniqueName: \"kubernetes.io/projected/2921d3c0-917b-4902-9263-5b667829adfc-kube-api-access-6bhns\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.334223 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.334231    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2921d3c0-917b-4902-9263-5b667829adfc-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:33:49.630774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.630684    5085 generic.go:356] "Generic (PLEG): container finished" podID="2921d3c0-917b-4902-9263-5b667829adfc" containerID="d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811" exitCode=0
Oct 30 20:33:49.630774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.630731    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"
Oct 30 20:33:49.630774 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.630765    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" event={"ID":"2921d3c0-917b-4902-9263-5b667829adfc","Type":"ContainerDied","Data":"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"}
Oct 30 20:33:49.631343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.630814    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn" event={"ID":"2921d3c0-917b-4902-9263-5b667829adfc","Type":"ContainerDied","Data":"215a28c4e0cbb2078973cbb15b092e82c545a3de4481696dc2291f6bbe2a4c36"}
Oct 30 20:33:49.631343 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.630852    5085 scope.go:122] "RemoveContainer" containerID="d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"
Oct 30 20:33:49.641796 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.641778    5085 scope.go:122] "RemoveContainer" containerID="d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"
Oct 30 20:33:49.642068 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:49.642045    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811\": container with ID starting with d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811 not found: ID does not exist" containerID="d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"
Oct 30 20:33:49.642140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.642075    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811"} err="failed to get container status \"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811\": rpc error: code = NotFound desc = could not find container \"d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811\": container with ID starting with d82d3e9e3cb817564f6a31ea0dfae347f1e8bbb24b53f0a98b845b33b57d1811 not found: ID does not exist"
Oct 30 20:33:49.789952 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.789913    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:33:49.798388 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:49.798355    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8d612c-7c64ffbbbc-kpxjn"]
Oct 30 20:33:50.633362 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:50.633323    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1212b686-9e45-4df9-832a-859013d89286" path="/var/lib/kubelet/pods/1212b686-9e45-4df9-832a-859013d89286/volumes"
Oct 30 20:33:50.633866 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:50.633801    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2921d3c0-917b-4902-9263-5b667829adfc" path="/var/lib/kubelet/pods/2921d3c0-917b-4902-9263-5b667829adfc/volumes"
Oct 30 20:33:51.940275 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:33:51.940238    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:33:57.398652 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:33:57.398611    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:34:05.423716 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:05.423684    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:34:05.533848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:05.533804    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:34:10.424533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:10.424491    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing" containerID="cri-o://d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe" gracePeriod=25
Oct 30 20:34:10.534160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:10.534125    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing" containerID="cri-o://375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c" gracePeriod=25
Oct 30 20:34:12.189295 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:12.189262    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:34:12.299396 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:12.299348    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:34:16.807837 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:16.807798    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:16.808252 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:16.807802    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:17.190160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:17.190113    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing" containerID="cri-o://042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f" gracePeriod=25
Oct 30 20:34:17.300328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:17.300282    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing" containerID="cri-o://c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12" gracePeriod=25
Oct 30 20:34:21.939712 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:21.939675    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:34:22.885491 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:22.885454    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:22.886657 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:22.886629    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:26.807341 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:26.807301    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:26.807852 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:26.807383    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:26.873209 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:26.873169    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:34:26.873464 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:26.873413    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" podUID="0661eb63-fc1b-4a03-8c02-a80320eb725d" containerName="maas-api" containerID="cri-o://6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6" gracePeriod=30
Oct 30 20:34:27.134104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.134074    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:34:27.292946 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.292916    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") pod \"0661eb63-fc1b-4a03-8c02-a80320eb725d\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") "
Oct 30 20:34:27.293134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.292967    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\") pod \"0661eb63-fc1b-4a03-8c02-a80320eb725d\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") "
Oct 30 20:34:27.293344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.293324    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\") pod \"0661eb63-fc1b-4a03-8c02-a80320eb725d\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") "
Oct 30 20:34:27.293446 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.293382    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca" pod "0661eb63-fc1b-4a03-8c02-a80320eb725d" (UID: "0661eb63-fc1b-4a03-8c02-a80320eb725d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:27.293521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.293468    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\" (UniqueName: \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\") pod \"0661eb63-fc1b-4a03-8c02-a80320eb725d\" (UID: \"0661eb63-fc1b-4a03-8c02-a80320eb725d\") "
Oct 30 20:34:27.294058 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.294035    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0661eb63-fc1b-4a03-8c02-a80320eb725d-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:27.296207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.296179    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls" pod "0661eb63-fc1b-4a03-8c02-a80320eb725d" (UID: "0661eb63-fc1b-4a03-8c02-a80320eb725d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:27.296318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.296191    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs" pod "0661eb63-fc1b-4a03-8c02-a80320eb725d" (UID: "0661eb63-fc1b-4a03-8c02-a80320eb725d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:27.297443 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.297414    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl" pod "0661eb63-fc1b-4a03-8c02-a80320eb725d" (UID: "0661eb63-fc1b-4a03-8c02-a80320eb725d"). InnerVolumeSpecName "kube-api-access-vfqcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:27.395411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.395321    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqcl\" (UniqueName: \"kubernetes.io/projected/0661eb63-fc1b-4a03-8c02-a80320eb725d-kube-api-access-vfqcl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:27.395411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.395354    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:27.395411 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.395364    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0661eb63-fc1b-4a03-8c02-a80320eb725d-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:27.917731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.917655    5085 generic.go:356] "Generic (PLEG): container finished" podID="0661eb63-fc1b-4a03-8c02-a80320eb725d" containerID="6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6" exitCode=0
Oct 30 20:34:27.917731 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.917702    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"
Oct 30 20:34:27.918133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.917742    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" event={"ID":"0661eb63-fc1b-4a03-8c02-a80320eb725d","Type":"ContainerDied","Data":"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"}
Oct 30 20:34:27.918133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.917796    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd" event={"ID":"0661eb63-fc1b-4a03-8c02-a80320eb725d","Type":"ContainerDied","Data":"07b3e10fefde29b6bcb5479733fdb01ef66bf1ad7f56b96e77922c5781cfe33d"}
Oct 30 20:34:27.918133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.917821    5085 scope.go:122] "RemoveContainer" containerID="6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"
Oct 30 20:34:27.928956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.928939    5085 scope.go:122] "RemoveContainer" containerID="6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"
Oct 30 20:34:27.929241 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:27.929221    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6\": container with ID starting with 6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6 not found: ID does not exist" containerID="6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"
Oct 30 20:34:27.929296 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:27.929249    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6"} err="failed to get container status \"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6\": rpc error: code = NotFound desc = could not find container \"6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6\": container with ID starting with 6bacdfbc30a9c8838b4379396958571182d003d2c383585d2492a1545607a0a6 not found: ID does not exist"
Oct 30 20:34:28.076111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:28.076076    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:34:28.124026 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:28.123988    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-487c5f-5fd5bddd85-q7wtd"]
Oct 30 20:34:28.631497 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:28.631458    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0661eb63-fc1b-4a03-8c02-a80320eb725d" path="/var/lib/kubelet/pods/0661eb63-fc1b-4a03-8c02-a80320eb725d/volumes"
Oct 30 20:34:30.698165 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.698097    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:34:30.727198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.727162    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\") pod \"989f3472-ef03-4865-8d85-7e5388757631\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") "
Oct 30 20:34:30.727345 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.727307    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\" (UniqueName: \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\") pod \"989f3472-ef03-4865-8d85-7e5388757631\" (UID: \"989f3472-ef03-4865-8d85-7e5388757631\") "
Oct 30 20:34:30.727543 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.727519    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume" pod "989f3472-ef03-4865-8d85-7e5388757631" (UID: "989f3472-ef03-4865-8d85-7e5388757631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:30.728107 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.728082    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989f3472-ef03-4865-8d85-7e5388757631-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:30.730140 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.730108    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9" pod "989f3472-ef03-4865-8d85-7e5388757631" (UID: "989f3472-ef03-4865-8d85-7e5388757631"). InnerVolumeSpecName "kube-api-access-bvmz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:30.786831 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.786803    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:34:30.828491 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.828463    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\") pod \"00a79f69-2a85-4fea-8037-aa339780c68d\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") "
Oct 30 20:34:30.828666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.828595    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\" (UniqueName: \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\") pod \"00a79f69-2a85-4fea-8037-aa339780c68d\" (UID: \"00a79f69-2a85-4fea-8037-aa339780c68d\") "
Oct 30 20:34:30.828862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.828841    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume" pod "00a79f69-2a85-4fea-8037-aa339780c68d" (UID: "00a79f69-2a85-4fea-8037-aa339780c68d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:30.828862 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.828856    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvmz9\" (UniqueName: \"kubernetes.io/projected/989f3472-ef03-4865-8d85-7e5388757631-kube-api-access-bvmz9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:30.830639 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.830621    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg" pod "00a79f69-2a85-4fea-8037-aa339780c68d" (UID: "00a79f69-2a85-4fea-8037-aa339780c68d"). InnerVolumeSpecName "kube-api-access-wh5hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:30.929694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.929665    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5hg\" (UniqueName: \"kubernetes.io/projected/00a79f69-2a85-4fea-8037-aa339780c68d-kube-api-access-wh5hg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:30.929694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.929691    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a79f69-2a85-4fea-8037-aa339780c68d-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:30.946787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.946749    5085 generic.go:356] "Generic (PLEG): container finished" podID="989f3472-ef03-4865-8d85-7e5388757631" containerID="d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe" exitCode=0
Oct 30 20:34:30.946957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.946797    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"
Oct 30 20:34:30.946957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.946840    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" event={"ID":"989f3472-ef03-4865-8d85-7e5388757631","Type":"ContainerDied","Data":"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"}
Oct 30 20:34:30.946957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.946914    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn" event={"ID":"989f3472-ef03-4865-8d85-7e5388757631","Type":"ContainerDied","Data":"2e71e30e7b7b8194008827f0571d2f00294c171ce357c1bca218b80c859b6c10"}
Oct 30 20:34:30.946957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.946946    5085 scope.go:122] "RemoveContainer" containerID="d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"
Oct 30 20:34:30.949331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.948818    5085 generic.go:356] "Generic (PLEG): container finished" podID="00a79f69-2a85-4fea-8037-aa339780c68d" containerID="375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c" exitCode=0
Oct 30 20:34:30.949331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.948897    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" event={"ID":"00a79f69-2a85-4fea-8037-aa339780c68d","Type":"ContainerDied","Data":"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"}
Oct 30 20:34:30.949331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.948933    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6" event={"ID":"00a79f69-2a85-4fea-8037-aa339780c68d","Type":"ContainerDied","Data":"8e5c4e911ea54a5626717f324172bd1427ea06ede56e181825f90bba0f7d3737"}
Oct 30 20:34:30.949331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.948993    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"
Oct 30 20:34:30.960935 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.960914    5085 scope.go:122] "RemoveContainer" containerID="d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"
Oct 30 20:34:30.961297 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:30.961273    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe\": container with ID starting with d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe not found: ID does not exist" containerID="d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"
Oct 30 20:34:30.961355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.961309    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe"} err="failed to get container status \"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe\": rpc error: code = NotFound desc = could not find container \"d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe\": container with ID starting with d92ebd9fd3d10d490a169de12f6f19a9e77e858060beccf1235a3e7f7305d0fe not found: ID does not exist"
Oct 30 20:34:30.961355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.961333    5085 scope.go:122] "RemoveContainer" containerID="375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"
Oct 30 20:34:30.973724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.973702    5085 scope.go:122] "RemoveContainer" containerID="375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"
Oct 30 20:34:30.974027 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:30.974000    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c\": container with ID starting with 375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c not found: ID does not exist" containerID="375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"
Oct 30 20:34:30.974123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:30.974036    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c"} err="failed to get container status \"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c\": rpc error: code = NotFound desc = could not find container \"375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c\": container with ID starting with 375a1961331c886b2e055b3aaea0ba51160c9d8de0578556933bcc0822f9ba0c not found: ID does not exist"
Oct 30 20:34:31.111332 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:31.111284    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:34:31.114911 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:31.114886    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-487c5f-7875df5768-7lbcn"]
Oct 30 20:34:31.231977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:31.231888    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:34:31.235027 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:31.235004    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-487c5f-8498b9b5bc-vmsh6"]
Oct 30 20:34:32.631459 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:32.631422    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" path="/var/lib/kubelet/pods/00a79f69-2a85-4fea-8037-aa339780c68d/volumes"
Oct 30 20:34:32.631914 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:32.631898    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989f3472-ef03-4865-8d85-7e5388757631" path="/var/lib/kubelet/pods/989f3472-ef03-4865-8d85-7e5388757631/volumes"
Oct 30 20:34:32.885089 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:32.885000    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:32.886846 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:32.886814    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:34:37.473683 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.473644    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:34:37.501092 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.501060    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\") pod \"7be81ba1-2653-4ca0-aece-ea26849bd079\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") "
Oct 30 20:34:37.501263 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.501121    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\" (UniqueName: \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\") pod \"7be81ba1-2653-4ca0-aece-ea26849bd079\" (UID: \"7be81ba1-2653-4ca0-aece-ea26849bd079\") "
Oct 30 20:34:37.501826 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.501783    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume" pod "7be81ba1-2653-4ca0-aece-ea26849bd079" (UID: "7be81ba1-2653-4ca0-aece-ea26849bd079"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:37.504206 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.504175    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff" pod "7be81ba1-2653-4ca0-aece-ea26849bd079" (UID: "7be81ba1-2653-4ca0-aece-ea26849bd079"). InnerVolumeSpecName "kube-api-access-9rmff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:37.561190 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.561167    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:34:37.602732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.602705    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\") pod \"dad19f68-49ab-42db-82f6-6cc8b2111f47\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") "
Oct 30 20:34:37.602870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.602832    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\" (UniqueName: \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\") pod \"dad19f68-49ab-42db-82f6-6cc8b2111f47\" (UID: \"dad19f68-49ab-42db-82f6-6cc8b2111f47\") "
Oct 30 20:34:37.603096 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.603038    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume" pod "dad19f68-49ab-42db-82f6-6cc8b2111f47" (UID: "dad19f68-49ab-42db-82f6-6cc8b2111f47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:37.603377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.603356    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7be81ba1-2653-4ca0-aece-ea26849bd079-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:37.603465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.603381    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmff\" (UniqueName: \"kubernetes.io/projected/7be81ba1-2653-4ca0-aece-ea26849bd079-kube-api-access-9rmff\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:37.603465 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.603396    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad19f68-49ab-42db-82f6-6cc8b2111f47-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:37.604966 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.604943    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz" pod "dad19f68-49ab-42db-82f6-6cc8b2111f47" (UID: "dad19f68-49ab-42db-82f6-6cc8b2111f47"). InnerVolumeSpecName "kube-api-access-7p8lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:37.705134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.705034    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8lz\" (UniqueName: \"kubernetes.io/projected/dad19f68-49ab-42db-82f6-6cc8b2111f47-kube-api-access-7p8lz\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:37.998222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.998135    5085 generic.go:356] "Generic (PLEG): container finished" podID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerID="c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12" exitCode=0
Oct 30 20:34:37.998222 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.998194    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"
Oct 30 20:34:37.998451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.998223    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" event={"ID":"dad19f68-49ab-42db-82f6-6cc8b2111f47","Type":"ContainerDied","Data":"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"}
Oct 30 20:34:37.998451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.998310    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5" event={"ID":"dad19f68-49ab-42db-82f6-6cc8b2111f47","Type":"ContainerDied","Data":"ea53b19ecbf069f66ed299980b50f6281c816d20d4f51676f98b3fcd052d9299"}
Oct 30 20:34:37.998451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.998329    5085 scope.go:122] "RemoveContainer" containerID="c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"
Oct 30 20:34:37.999675 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.999652    5085 generic.go:356] "Generic (PLEG): container finished" podID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerID="042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f" exitCode=0
Oct 30 20:34:37.999787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.999710    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"
Oct 30 20:34:37.999787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.999751    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" event={"ID":"7be81ba1-2653-4ca0-aece-ea26849bd079","Type":"ContainerDied","Data":"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"}
Oct 30 20:34:37.999887 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:37.999798    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh" event={"ID":"7be81ba1-2653-4ca0-aece-ea26849bd079","Type":"ContainerDied","Data":"674ea0106a62ee907ce7ef2a73a87e0245706ab35772dc0d16fc109c2a699b81"}
Oct 30 20:34:38.009072 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.009055    5085 scope.go:122] "RemoveContainer" containerID="c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"
Oct 30 20:34:38.009402 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:38.009381    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12\": container with ID starting with c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12 not found: ID does not exist" containerID="c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"
Oct 30 20:34:38.009487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.009414    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12"} err="failed to get container status \"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12\": rpc error: code = NotFound desc = could not find container \"c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12\": container with ID starting with c7545df2dcf1853518afd7d9927594262558ace540ef29caea01a3ab5048ba12 not found: ID does not exist"
Oct 30 20:34:38.009487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.009437    5085 scope.go:122] "RemoveContainer" containerID="042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"
Oct 30 20:34:38.018540 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.018524    5085 scope.go:122] "RemoveContainer" containerID="042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"
Oct 30 20:34:38.018790 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:38.018770    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f\": container with ID starting with 042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f not found: ID does not exist" containerID="042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"
Oct 30 20:34:38.018866 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.018800    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f"} err="failed to get container status \"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f\": rpc error: code = NotFound desc = could not find container \"042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f\": container with ID starting with 042a5373172e2cd8e5446ed9bf29e7c24138753ba1f29f167b1f40a42f3a421f not found: ID does not exist"
Oct 30 20:34:38.155852 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.155811    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:34:38.161903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.161881    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cafbb2-549d67b8c5-qg7k5"]
Oct 30 20:34:38.265037 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.264960    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:34:38.268053 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.268028    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-cafbb2-68549d456b-t2xwh"]
Oct 30 20:34:38.632325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.632238    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" path="/var/lib/kubelet/pods/7be81ba1-2653-4ca0-aece-ea26849bd079/volumes"
Oct 30 20:34:38.632899 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:38.632869    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" path="/var/lib/kubelet/pods/dad19f68-49ab-42db-82f6-6cc8b2111f47/volumes"
Oct 30 20:34:39.288739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.288708    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:34:39.288936 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.288893    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" podUID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" containerName="maas-api" containerID="cri-o://f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12" gracePeriod=30
Oct 30 20:34:39.553243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.553186    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:34:39.626982 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.626954    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\") pod \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") "
Oct 30 20:34:39.627158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.627059    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\") pod \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") "
Oct 30 20:34:39.627158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.627090    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") pod \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") "
Oct 30 20:34:39.627158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.627127    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\" (UniqueName: \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\") pod \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\" (UID: \"e5389f98-ee52-4bee-b393-cdb7ad0836f2\") "
Oct 30 20:34:39.627429 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.627407    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca" pod "e5389f98-ee52-4bee-b393-cdb7ad0836f2" (UID: "e5389f98-ee52-4bee-b393-cdb7ad0836f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:34:39.629048 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.629020    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs" pod "e5389f98-ee52-4bee-b393-cdb7ad0836f2" (UID: "e5389f98-ee52-4bee-b393-cdb7ad0836f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:39.629268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.629246    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7" pod "e5389f98-ee52-4bee-b393-cdb7ad0836f2" (UID: "e5389f98-ee52-4bee-b393-cdb7ad0836f2"). InnerVolumeSpecName "kube-api-access-v5pl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:34:39.629332 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.629280    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls" pod "e5389f98-ee52-4bee-b393-cdb7ad0836f2" (UID: "e5389f98-ee52-4bee-b393-cdb7ad0836f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:34:39.729067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.729037    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:39.729067 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.729065    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pl7\" (UniqueName: \"kubernetes.io/projected/e5389f98-ee52-4bee-b393-cdb7ad0836f2-kube-api-access-v5pl7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:39.729485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.729078    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5389f98-ee52-4bee-b393-cdb7ad0836f2-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:39.729485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:39.729091    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5389f98-ee52-4bee-b393-cdb7ad0836f2-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:34:40.017432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.017395    5085 generic.go:356] "Generic (PLEG): container finished" podID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" containerID="f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12" exitCode=0
Oct 30 20:34:40.017700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.017439    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"
Oct 30 20:34:40.017700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.017454    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" event={"ID":"e5389f98-ee52-4bee-b393-cdb7ad0836f2","Type":"ContainerDied","Data":"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"}
Oct 30 20:34:40.017700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.017500    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx" event={"ID":"e5389f98-ee52-4bee-b393-cdb7ad0836f2","Type":"ContainerDied","Data":"5f949d7c7f033a8933d6620914bf162a84dfa1940c33929f1c4c487069b1e397"}
Oct 30 20:34:40.017700 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.017521    5085 scope.go:122] "RemoveContainer" containerID="f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"
Oct 30 20:34:40.029132 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.029114    5085 scope.go:122] "RemoveContainer" containerID="f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"
Oct 30 20:34:40.029405 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:40.029389    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12\": container with ID starting with f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12 not found: ID does not exist" containerID="f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"
Oct 30 20:34:40.029456 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.029413    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12"} err="failed to get container status \"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12\": rpc error: code = NotFound desc = could not find container \"f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12\": container with ID starting with f8e669494847db85598c85dbee283e71e4fc5011ee1ce9ad471431dc8ba1de12 not found: ID does not exist"
Oct 30 20:34:40.177546 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.177504    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:34:40.181006 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.180981    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cafbb2-c9b66b5f4-5pxtx"]
Oct 30 20:34:40.632370 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:40.632338    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" path="/var/lib/kubelet/pods/e5389f98-ee52-4bee-b393-cdb7ad0836f2/volumes"
Oct 30 20:34:49.252952 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:49.252917    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:34:49.406451 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:49.406411    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:34:51.297425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:51.297387    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:34:51.586624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:51.586525    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:34:51.598441 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:51.598414    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:34:51.943588 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:34:51.943531    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:34:54.253790 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:54.253751    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing" containerID="cri-o://5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75" gracePeriod=25
Oct 30 20:34:54.407767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:34:54.407730    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing" containerID="cri-o://73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac" gracePeriod=25
Oct 30 20:35:00.807636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:00.807594    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:00.807636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:00.807628    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:01.686154 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:01.686124    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:35:01.796898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:01.796862    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:35:06.686638 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:06.686591    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing" containerID="cri-o://8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5" gracePeriod=25
Oct 30 20:35:06.798071 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:06.798035    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing" containerID="cri-o://fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b" gracePeriod=25
Oct 30 20:35:06.867228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:06.867193    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:06.867410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:06.867200    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:10.807183 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:10.807145    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:10.807624 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:10.807147    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:14.546054 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.546029    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:35:14.632287 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.632258    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\" (UniqueName: \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\") pod \"c5913b93-1358-42e4-a311-6a1170a00a41\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") "
Oct 30 20:35:14.632482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.632323    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\") pod \"c5913b93-1358-42e4-a311-6a1170a00a41\" (UID: \"c5913b93-1358-42e4-a311-6a1170a00a41\") "
Oct 30 20:35:14.633109 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.633079    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume" pod "c5913b93-1358-42e4-a311-6a1170a00a41" (UID: "c5913b93-1358-42e4-a311-6a1170a00a41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:14.634650 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.634617    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr" pod "c5913b93-1358-42e4-a311-6a1170a00a41" (UID: "c5913b93-1358-42e4-a311-6a1170a00a41"). InnerVolumeSpecName "kube-api-access-78npr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:14.680489 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.680464    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:35:14.734004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.733976    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78npr\" (UniqueName: \"kubernetes.io/projected/c5913b93-1358-42e4-a311-6a1170a00a41-kube-api-access-78npr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:14.734004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.734001    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5913b93-1358-42e4-a311-6a1170a00a41-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:14.835018 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.834928    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\") pod \"2e6f6784-eb00-4cad-b0b3-4365f8446172\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") "
Oct 30 20:35:14.835197 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.835026    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\" (UniqueName: \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\") pod \"2e6f6784-eb00-4cad-b0b3-4365f8446172\" (UID: \"2e6f6784-eb00-4cad-b0b3-4365f8446172\") "
Oct 30 20:35:14.835302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.835277    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume" pod "2e6f6784-eb00-4cad-b0b3-4365f8446172" (UID: "2e6f6784-eb00-4cad-b0b3-4365f8446172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:14.835500 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.835482    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6f6784-eb00-4cad-b0b3-4365f8446172-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:14.837233 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.837209    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp" pod "2e6f6784-eb00-4cad-b0b3-4365f8446172" (UID: "2e6f6784-eb00-4cad-b0b3-4365f8446172"). InnerVolumeSpecName "kube-api-access-kt4vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:14.936326 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:14.936296    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4vp\" (UniqueName: \"kubernetes.io/projected/2e6f6784-eb00-4cad-b0b3-4365f8446172-kube-api-access-kt4vp\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:15.260830 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.260793    5085 generic.go:356] "Generic (PLEG): container finished" podID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerID="73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac" exitCode=0
Oct 30 20:35:15.261038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.260855    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"
Oct 30 20:35:15.261038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.260862    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" event={"ID":"2e6f6784-eb00-4cad-b0b3-4365f8446172","Type":"ContainerDied","Data":"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"}
Oct 30 20:35:15.261038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.260941    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq" event={"ID":"2e6f6784-eb00-4cad-b0b3-4365f8446172","Type":"ContainerDied","Data":"ffa89380fc6e90c3cf102e86fc3a4095afe3fb056ba9fde43b01907c8e4b37d9"}
Oct 30 20:35:15.261038 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.260963    5085 scope.go:122] "RemoveContainer" containerID="73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"
Oct 30 20:35:15.262531 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.262508    5085 generic.go:356] "Generic (PLEG): container finished" podID="c5913b93-1358-42e4-a311-6a1170a00a41" containerID="5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75" exitCode=0
Oct 30 20:35:15.262666 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.262600    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"
Oct 30 20:35:15.262753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.262600    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" event={"ID":"c5913b93-1358-42e4-a311-6a1170a00a41","Type":"ContainerDied","Data":"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"}
Oct 30 20:35:15.262753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.262707    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s" event={"ID":"c5913b93-1358-42e4-a311-6a1170a00a41","Type":"ContainerDied","Data":"545a23c564c1feca67288110fade14a3bd9356ad9eb021ef339f252186948f42"}
Oct 30 20:35:15.287891 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.287871    5085 scope.go:122] "RemoveContainer" containerID="73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"
Oct 30 20:35:15.288200 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:15.288180    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac\": container with ID starting with 73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac not found: ID does not exist" containerID="73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"
Oct 30 20:35:15.288281 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.288208    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac"} err="failed to get container status \"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac\": rpc error: code = NotFound desc = could not find container \"73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac\": container with ID starting with 73317b80ad0054aed96fc892d599e951ef785faf4d2f2387fb04ebfe03c6d0ac not found: ID does not exist"
Oct 30 20:35:15.288281 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.288224    5085 scope.go:122] "RemoveContainer" containerID="5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"
Oct 30 20:35:15.297360 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.297342    5085 scope.go:122] "RemoveContainer" containerID="5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"
Oct 30 20:35:15.297639 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:15.297619    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75\": container with ID starting with 5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75 not found: ID does not exist" containerID="5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"
Oct 30 20:35:15.297704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.297648    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75"} err="failed to get container status \"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75\": rpc error: code = NotFound desc = could not find container \"5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75\": container with ID starting with 5eace10a6483d7ef3e5aae75fa0a196106e50067b41935e1b7e8398473a4ce75 not found: ID does not exist"
Oct 30 20:35:15.421002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.420970    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:35:15.429417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.429389    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fe49e7-6bb448b78c-v4rcq"]
Oct 30 20:35:15.534237 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.534159    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:35:15.539322 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:15.539295    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-fe49e7-7864f67bd-mtx2s"]
Oct 30 20:35:16.632230 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:16.632195    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" path="/var/lib/kubelet/pods/2e6f6784-eb00-4cad-b0b3-4365f8446172/volumes"
Oct 30 20:35:16.632688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:16.632672    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" path="/var/lib/kubelet/pods/c5913b93-1358-42e4-a311-6a1170a00a41/volumes"
Oct 30 20:35:16.866981 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:16.866945    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:16.867161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:16.866951    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:35:17.224116 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224080    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:35:17.224816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224787    5085 cpu_manager.go:414] "containerMap: removing container" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing"
Oct 30 20:35:17.224816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224808    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing"
Oct 30 20:35:17.224816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224820    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5098658d-9bc1-41cd-a002-e60fd349937c" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224827    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5098658d-9bc1-41cd-a002-e60fd349937c" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224846    5085 cpu_manager.go:414] "containerMap: removing container" podUID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224852    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224863    5085 cpu_manager.go:414] "containerMap: removing container" podUID="bfec0531-34f8-4963-95f6-18e1d1d757e9" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224870    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfec0531-34f8-4963-95f6-18e1d1d757e9" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224894    5085 cpu_manager.go:414] "containerMap: removing container" podUID="1212b686-9e45-4df9-832a-859013d89286" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224901    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1212b686-9e45-4df9-832a-859013d89286" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224914    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224920    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224930    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224937    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224952    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2921d3c0-917b-4902-9263-5b667829adfc" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224959    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2921d3c0-917b-4902-9263-5b667829adfc" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224967    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0661eb63-fc1b-4a03-8c02-a80320eb725d" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224974    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0661eb63-fc1b-4a03-8c02-a80320eb725d" containerName="maas-api"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224989    5085 cpu_manager.go:414] "containerMap: removing container" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.224996    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225006    5085 cpu_manager.go:414] "containerMap: removing container" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225014    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225041    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing"
Oct 30 20:35:17.225077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225049    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225214    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="989f3472-ef03-4865-8d85-7e5388757631" containerName="payload-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225230    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5098658d-9bc1-41cd-a002-e60fd349937c" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225241    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5389f98-ee52-4bee-b393-cdb7ad0836f2" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225253    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfec0531-34f8-4963-95f6-18e1d1d757e9" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225266    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="1212b686-9e45-4df9-832a-859013d89286" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225275    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be81ba1-2653-4ca0-aece-ea26849bd079" containerName="payload-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225284    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6f6784-eb00-4cad-b0b3-4365f8446172" containerName="payload-pre-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225294    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2921d3c0-917b-4902-9263-5b667829adfc" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225301    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0661eb63-fc1b-4a03-8c02-a80320eb725d" containerName="maas-api"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225309    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a79f69-2a85-4fea-8037-aa339780c68d" containerName="payload-pre-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225317    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad19f68-49ab-42db-82f6-6cc8b2111f47" containerName="payload-pre-processing"
Oct 30 20:35:17.226155 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.225328    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5913b93-1358-42e4-a311-6a1170a00a41" containerName="payload-processing"
Oct 30 20:35:17.231381 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.231359    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.283069 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.283039    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-46dadd-serving-cert\""
Oct 30 20:35:17.332143 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.332111    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:35:17.363375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.363342    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.363551 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.363427    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.363551 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.363506    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv7km\" (UniqueName: \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.363659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.363553    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.464893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.464853    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.465117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.464963    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.465117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.465056    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv7km\" (UniqueName: \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.465117 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.465113    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.465499 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:17.465440    5085 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-serving-cert: secret "maas-api-e2e-extmod-a-46dadd-serving-cert" not found
Oct 30 20:35:17.465645 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:17.465539    5085 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls podName:002593c7-ca36-4d55-9cbf-270d2417426c nodeName:}" failed. No retries permitted until 2026-10-30 20:35:17.96551849 +0000 UTC m=+4227.469922229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls") pod "maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" (UID: "002593c7-ca36-4d55-9cbf-270d2417426c") : secret "maas-api-e2e-extmod-a-46dadd-serving-cert" not found
Oct 30 20:35:17.465793 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.465762    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.468775 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.468746    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.529023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.528952    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv7km\" (UniqueName: \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.971199 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.971168    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:17.973510 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:17.973489    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:18.145870 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:18.145835    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:18.520707 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:18.520675    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:35:18.521731 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:35:18.521699    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod002593c7_ca36_4d55_9cbf_270d2417426c.slice/crio-9b45a7a9604254275ec01da2d869de7043e0826d21cfe765e4a46b2769738b0c WatchSource:0}: Error finding container 9b45a7a9604254275ec01da2d869de7043e0826d21cfe765e4a46b2769738b0c: Status 404 returned error can't find the container with id 9b45a7a9604254275ec01da2d869de7043e0826d21cfe765e4a46b2769738b0c
Oct 30 20:35:18.523225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:18.523207    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:35:19.291973 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:19.291933    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" event={"ID":"002593c7-ca36-4d55-9cbf-270d2417426c","Type":"ContainerStarted","Data":"9b45a7a9604254275ec01da2d869de7043e0826d21cfe765e4a46b2769738b0c"}
Oct 30 20:35:20.309601 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:20.309535    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" event={"ID":"002593c7-ca36-4d55-9cbf-270d2417426c","Type":"ContainerStarted","Data":"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"}
Oct 30 20:35:20.310080 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:20.309763    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:20.425482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:20.425415    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" podStartSLOduration=2.371816152 podStartE2EDuration="3.425400415s" totalImagesPullingTime="1.053584263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:35:20.422701404 +0000 UTC m=+4229.927105184" watchObservedRunningTime="2026-10-30 20:35:20.425400415 +0000 UTC m=+4229.929804206"
Oct 30 20:35:21.938838 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:21.938794    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:35:26.323581 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.323537    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:35:26.867497 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:35:26.867472    5085 logging.go:55] [core] [Channel #8048 SubChannel #8050]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.252:50052", ServerName: "10.132.0.252:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.252:50052: connect: connection refused"
Oct 30 20:35:26.964574 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.964543    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:35:26.976250 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.976225    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\" (UniqueName: \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\") pod \"5ac207a5-b4f6-4897-b724-5533aa03dac2\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") "
Oct 30 20:35:26.976402 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.976260    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\") pod \"5ac207a5-b4f6-4897-b724-5533aa03dac2\" (UID: \"5ac207a5-b4f6-4897-b724-5533aa03dac2\") "
Oct 30 20:35:26.976739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.976697    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume" pod "5ac207a5-b4f6-4897-b724-5533aa03dac2" (UID: "5ac207a5-b4f6-4897-b724-5533aa03dac2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:26.976903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.976872    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ac207a5-b4f6-4897-b724-5533aa03dac2-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:26.978482 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:26.978454    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94" pod "5ac207a5-b4f6-4897-b724-5533aa03dac2" (UID: "5ac207a5-b4f6-4897-b724-5533aa03dac2"). InnerVolumeSpecName "kube-api-access-4zs94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:27.065726 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.065701    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:35:27.077892 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.077866    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\") pod \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") "
Oct 30 20:35:27.078019 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.077980    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\" (UniqueName: \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\") pod \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\" (UID: \"04fda8e3-64e1-4da7-b2dd-9cb95994254b\") "
Oct 30 20:35:27.078299 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.078272    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume" pod "04fda8e3-64e1-4da7-b2dd-9cb95994254b" (UID: "04fda8e3-64e1-4da7-b2dd-9cb95994254b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:27.078452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.078436    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fda8e3-64e1-4da7-b2dd-9cb95994254b-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.078506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.078455    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zs94\" (UniqueName: \"kubernetes.io/projected/5ac207a5-b4f6-4897-b724-5533aa03dac2-kube-api-access-4zs94\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.079956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.079931    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq" pod "04fda8e3-64e1-4da7-b2dd-9cb95994254b" (UID: "04fda8e3-64e1-4da7-b2dd-9cb95994254b"). InnerVolumeSpecName "kube-api-access-rnphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:27.180039 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.180001    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnphq\" (UniqueName: \"kubernetes.io/projected/04fda8e3-64e1-4da7-b2dd-9cb95994254b-kube-api-access-rnphq\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:27.358348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.358251    5085 generic.go:356] "Generic (PLEG): container finished" podID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerID="8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5" exitCode=0
Oct 30 20:35:27.358348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.358303    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"
Oct 30 20:35:27.359139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.358298    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" event={"ID":"5ac207a5-b4f6-4897-b724-5533aa03dac2","Type":"ContainerDied","Data":"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"}
Oct 30 20:35:27.359139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.358427    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" event={"ID":"5ac207a5-b4f6-4897-b724-5533aa03dac2","Type":"ContainerDied","Data":"0400bdac5c6978b492fd91cab4448018ee72212e7404af917fdd95e59cf524eb"}
Oct 30 20:35:27.359139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.358461    5085 scope.go:122] "RemoveContainer" containerID="8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"
Oct 30 20:35:27.360953 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.360930    5085 generic.go:356] "Generic (PLEG): container finished" podID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerID="fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b" exitCode=0
Oct 30 20:35:27.361099 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.361065    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" event={"ID":"04fda8e3-64e1-4da7-b2dd-9cb95994254b","Type":"ContainerDied","Data":"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"}
Oct 30 20:35:27.361168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.361099    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" event={"ID":"04fda8e3-64e1-4da7-b2dd-9cb95994254b","Type":"ContainerDied","Data":"0d5b184baba25155dac95cc8f3f39b61a6138ec0822a0905cab42efb3e7d6e97"}
Oct 30 20:35:27.361168 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.361152    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"
Oct 30 20:35:27.371425 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.371404    5085 scope.go:122] "RemoveContainer" containerID="8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"
Oct 30 20:35:27.371762 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:27.371740    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5\": container with ID starting with 8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5 not found: ID does not exist" containerID="8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"
Oct 30 20:35:27.371861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.371772    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5"} err="failed to get container status \"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5\": rpc error: code = NotFound desc = could not find container \"8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5\": container with ID starting with 8bd105e05f2b92fae47e94567287efc09ab2e1e3402752f74f2936a7a048cad5 not found: ID does not exist"
Oct 30 20:35:27.371861 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.371794    5085 scope.go:122] "RemoveContainer" containerID="fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"
Oct 30 20:35:27.380557 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.380542    5085 scope.go:122] "RemoveContainer" containerID="fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"
Oct 30 20:35:27.380869 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:27.380850    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b\": container with ID starting with fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b not found: ID does not exist" containerID="fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"
Oct 30 20:35:27.380913 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.380876    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b"} err="failed to get container status \"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b\": rpc error: code = NotFound desc = could not find container \"fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b\": container with ID starting with fe2cf4a7b66087579c7426eeb9953d409ed5c2d978b8b270e57905fc4f87910b not found: ID does not exist"
Oct 30 20:35:27.517333 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.517293    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:35:27.522613 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.522581    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k"]
Oct 30 20:35:27.626773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.626693    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:35:27.635995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.635971    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w"]
Oct 30 20:35:27.867969 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.867930    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-b37684-77fcd69c56-6ns9w" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.253:50052\" within 1s: context deadline exceeded"
Oct 30 20:35:27.868147 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:27.867930    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-b37684-5674b77867-mv74k" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.252:50052\" within 1s: context deadline exceeded"
Oct 30 20:35:27.868147 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:35:27.868014    5085 logging.go:55] [core] [Channel #8049 SubChannel #8051]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.253:50052", ServerName: "10.132.0.253:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.253:50052: operation was canceled"
Oct 30 20:35:28.632061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:28.632028    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" path="/var/lib/kubelet/pods/04fda8e3-64e1-4da7-b2dd-9cb95994254b/volumes"
Oct 30 20:35:28.632463 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:28.632449    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" path="/var/lib/kubelet/pods/5ac207a5-b4f6-4897-b724-5533aa03dac2/volumes"
Oct 30 20:35:37.242732 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:37.242693    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 20:35:38.634771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:38.634739    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:35:39.467506 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.467473    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:35:39.467882 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.467827    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" containerName="maas-api" containerID="cri-o://a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00" gracePeriod=30
Oct 30 20:35:39.741207 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.741178    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:35:39.817347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817308    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\") pod \"af006b12-d6df-4191-b47a-7f3100f1416b\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") "
Oct 30 20:35:39.817548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817355    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\") pod \"af006b12-d6df-4191-b47a-7f3100f1416b\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") "
Oct 30 20:35:39.817548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817466    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") pod \"af006b12-d6df-4191-b47a-7f3100f1416b\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") "
Oct 30 20:35:39.817548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817527    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\" (UniqueName: \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\") pod \"af006b12-d6df-4191-b47a-7f3100f1416b\" (UID: \"af006b12-d6df-4191-b47a-7f3100f1416b\") "
Oct 30 20:35:39.817767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817722    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca" pod "af006b12-d6df-4191-b47a-7f3100f1416b" (UID: "af006b12-d6df-4191-b47a-7f3100f1416b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:39.818007 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.817989    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af006b12-d6df-4191-b47a-7f3100f1416b-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:39.819623 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.819592    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs" pod "af006b12-d6df-4191-b47a-7f3100f1416b" (UID: "af006b12-d6df-4191-b47a-7f3100f1416b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:39.819742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.819644    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls" pod "af006b12-d6df-4191-b47a-7f3100f1416b" (UID: "af006b12-d6df-4191-b47a-7f3100f1416b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:39.819742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.819665    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v" pod "af006b12-d6df-4191-b47a-7f3100f1416b" (UID: "af006b12-d6df-4191-b47a-7f3100f1416b"). InnerVolumeSpecName "kube-api-access-4596v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:39.919595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.919525    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4596v\" (UniqueName: \"kubernetes.io/projected/af006b12-d6df-4191-b47a-7f3100f1416b-kube-api-access-4596v\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:39.919595 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.919596    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:39.919808 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:39.919615    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af006b12-d6df-4191-b47a-7f3100f1416b-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:40.449771 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.449734    5085 generic.go:356] "Generic (PLEG): container finished" podID="af006b12-d6df-4191-b47a-7f3100f1416b" containerID="a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00" exitCode=0
Oct 30 20:35:40.449977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.449778    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"
Oct 30 20:35:40.449977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.449819    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" event={"ID":"af006b12-d6df-4191-b47a-7f3100f1416b","Type":"ContainerDied","Data":"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"}
Oct 30 20:35:40.449977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.449873    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" event={"ID":"af006b12-d6df-4191-b47a-7f3100f1416b","Type":"ContainerDied","Data":"065a218983bf73f4c0b32dd93c05d22acd0bedb6d4b96422728fcc2d8ad055f6"}
Oct 30 20:35:40.449977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.449897    5085 scope.go:122] "RemoveContainer" containerID="a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"
Oct 30 20:35:40.460893 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.460875    5085 scope.go:122] "RemoveContainer" containerID="a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"
Oct 30 20:35:40.461159 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:40.461140    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00\": container with ID starting with a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00 not found: ID does not exist" containerID="a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"
Oct 30 20:35:40.461229 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.461167    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00"} err="failed to get container status \"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00\": rpc error: code = NotFound desc = could not find container \"a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00\": container with ID starting with a5abfe2149c46addbd38d6b0eff5f4bc1bca4a434f6763ae4e59ecbe88423f00 not found: ID does not exist"
Oct 30 20:35:40.610956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.610923    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:35:40.660578 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:40.660521    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q"]
Oct 30 20:35:42.632265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:42.632227    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" path="/var/lib/kubelet/pods/af006b12-d6df-4191-b47a-7f3100f1416b/volumes"
Oct 30 20:35:42.722780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:42.722732    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0e72bd-67b7dccf65-z452q" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.13:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:35:51.939746 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:51.939709    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:35:52.263218 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.263133    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" containerID="cri-o://541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b" gracePeriod=45
Oct 30 20:35:52.319328 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.319295    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": dial tcp 10.132.0.82:8000: connect: connection refused"
Oct 30 20:35:52.526872 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.526785    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 20:35:52.531367 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.531339    5085 generic.go:356] "Generic (PLEG): container finished" podID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerID="541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b" exitCode=0
Oct 30 20:35:52.531552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.531533    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" event={"ID":"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545","Type":"ContainerDied","Data":"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"}
Oct 30 20:35:52.531661 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.531590    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh" event={"ID":"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545","Type":"ContainerDied","Data":"bb081b3cd09d70dacae30753be8d926b3f2177d759a1ad6540003e71d9f16c42"}
Oct 30 20:35:52.531661 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.531615    5085 scope.go:122] "RemoveContainer" containerID="541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"
Oct 30 20:35:52.531784 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.531761    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"
Oct 30 20:35:52.542174 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.542151    5085 scope.go:122] "RemoveContainer" containerID="541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"
Oct 30 20:35:52.542442 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:52.542413    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b\": container with ID starting with 541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b not found: ID does not exist" containerID="541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"
Oct 30 20:35:52.542529 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.542448    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b"} err="failed to get container status \"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b\": rpc error: code = NotFound desc = could not find container \"541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b\": container with ID starting with 541c7c4a557464cd33b543252d834059b8d207dd3d5477310c1f8131808b563b not found: ID does not exist"
Oct 30 20:35:52.661382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661350    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661406    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661474    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661526    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\" (UniqueName: \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661601    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\" (UniqueName: \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661690    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\") pod \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\" (UID: \"a60f85e1-ae7b-4a9c-8f7b-65b740e6b545\") "
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661711    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.661745    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:52.662215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.662015    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:52.662783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.662372    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.662783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.662413    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.662783 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.662427    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.664488 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.664346    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:35:52.664488 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.664473    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "kube-api-access-jl249". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:52.665011 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.664991    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs" pod "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" (UID: "a60f85e1-ae7b-4a9c-8f7b-65b740e6b545"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:52.763943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.763908    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl249\" (UniqueName: \"kubernetes.io/projected/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-kube-api-access-jl249\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.763943 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.763942    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.764151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.763956    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:52.948758 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.948722    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 20:35:52.952974 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:52.952947    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ec609/simulator-w4-4ec609-kserve-f955958fb-9pggh"]
Oct 30 20:35:54.631104 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:54.631075    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" path="/var/lib/kubelet/pods/a60f85e1-ae7b-4a9c-8f7b-65b740e6b545/volumes"
Oct 30 20:35:57.223579 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.223533    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:35:57.570816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.570719    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" podUID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" containerName="maas-api" containerID="cri-o://71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0" gracePeriod=30
Oct 30 20:35:57.837154 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.837042    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:35:57.923232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923202    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\" (UniqueName: \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\") pod \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") "
Oct 30 20:35:57.923419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923241    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\") pod \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") "
Oct 30 20:35:57.923419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923293    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\") pod \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") "
Oct 30 20:35:57.923553 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923447    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") pod \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\" (UID: \"0b5b87b5-dec5-43f2-8d44-e595f3964ad1\") "
Oct 30 20:35:57.923725 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923704    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca" pod "0b5b87b5-dec5-43f2-8d44-e595f3964ad1" (UID: "0b5b87b5-dec5-43f2-8d44-e595f3964ad1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:35:57.924012 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.923994    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:57.925382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.925360    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l" pod "0b5b87b5-dec5-43f2-8d44-e595f3964ad1" (UID: "0b5b87b5-dec5-43f2-8d44-e595f3964ad1"). InnerVolumeSpecName "kube-api-access-6tp9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:35:57.925480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.925400    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs" pod "0b5b87b5-dec5-43f2-8d44-e595f3964ad1" (UID: "0b5b87b5-dec5-43f2-8d44-e595f3964ad1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:57.925480 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:57.925427    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls" pod "0b5b87b5-dec5-43f2-8d44-e595f3964ad1" (UID: "0b5b87b5-dec5-43f2-8d44-e595f3964ad1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:35:58.025298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.025264    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:58.025298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.025292    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tp9l\" (UniqueName: \"kubernetes.io/projected/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-kube-api-access-6tp9l\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:58.025298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.025303    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b5b87b5-dec5-43f2-8d44-e595f3964ad1-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:35:58.577791 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.577756    5085 generic.go:356] "Generic (PLEG): container finished" podID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" containerID="71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0" exitCode=0
Oct 30 20:35:58.578260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.577797    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"
Oct 30 20:35:58.578260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.577842    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" event={"ID":"0b5b87b5-dec5-43f2-8d44-e595f3964ad1","Type":"ContainerDied","Data":"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"}
Oct 30 20:35:58.578260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.577892    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc" event={"ID":"0b5b87b5-dec5-43f2-8d44-e595f3964ad1","Type":"ContainerDied","Data":"159223853caa038727ada49818e5ca5eb94d666c080ca47229b21a5b90aeb8aa"}
Oct 30 20:35:58.578260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.577915    5085 scope.go:122] "RemoveContainer" containerID="71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"
Oct 30 20:35:58.589065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.589044    5085 scope.go:122] "RemoveContainer" containerID="71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"
Oct 30 20:35:58.589342 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:35:58.589324    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0\": container with ID starting with 71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0 not found: ID does not exist" containerID="71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"
Oct 30 20:35:58.589390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.589348    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0"} err="failed to get container status \"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0\": rpc error: code = NotFound desc = could not find container \"71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0\": container with ID starting with 71627f8a4d361dde249c95deabf615ce6599989974a9169a3bdedc791a43ccd0 not found: ID does not exist"
Oct 30 20:35:58.742181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.742151    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:35:58.789129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:35:58.789093    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-fe49e7-5fdfc4b595-snmrc"]
Oct 30 20:36:00.631008 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:00.630973    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" path="/var/lib/kubelet/pods/0b5b87b5-dec5-43f2-8d44-e595f3964ad1/volumes"
Oct 30 20:36:09.585151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.585115    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:36:09.585740 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.585316    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" podUID="f1022e3f-ea50-4bca-961f-e8a71e390086" containerName="maas-api" containerID="cri-o://08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978" gracePeriod=30
Oct 30 20:36:09.881848 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.881823    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:36:09.954151 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.954123    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") pod \"f1022e3f-ea50-4bca-961f-e8a71e390086\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") "
Oct 30 20:36:09.954344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.954208    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\" (UniqueName: \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\") pod \"f1022e3f-ea50-4bca-961f-e8a71e390086\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") "
Oct 30 20:36:09.954344 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.954281    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\") pod \"f1022e3f-ea50-4bca-961f-e8a71e390086\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") "
Oct 30 20:36:09.954492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.954351    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\") pod \"f1022e3f-ea50-4bca-961f-e8a71e390086\" (UID: \"f1022e3f-ea50-4bca-961f-e8a71e390086\") "
Oct 30 20:36:09.954792 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.954763    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca" pod "f1022e3f-ea50-4bca-961f-e8a71e390086" (UID: "f1022e3f-ea50-4bca-961f-e8a71e390086"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:36:09.956298 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.956276    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls" pod "f1022e3f-ea50-4bca-961f-e8a71e390086" (UID: "f1022e3f-ea50-4bca-961f-e8a71e390086"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:09.956406 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.956321    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs" pod "f1022e3f-ea50-4bca-961f-e8a71e390086" (UID: "f1022e3f-ea50-4bca-961f-e8a71e390086"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:09.956469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:09.956401    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn" pod "f1022e3f-ea50-4bca-961f-e8a71e390086" (UID: "f1022e3f-ea50-4bca-961f-e8a71e390086"). InnerVolumeSpecName "kube-api-access-qhghn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:10.055884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.055847    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.055884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.055880    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1022e3f-ea50-4bca-961f-e8a71e390086-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.056099 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.055894    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1022e3f-ea50-4bca-961f-e8a71e390086-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.056099 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.055907    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhghn\" (UniqueName: \"kubernetes.io/projected/f1022e3f-ea50-4bca-961f-e8a71e390086-kube-api-access-qhghn\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:10.670361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.669871    5085 generic.go:356] "Generic (PLEG): container finished" podID="f1022e3f-ea50-4bca-961f-e8a71e390086" containerID="08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978" exitCode=0
Oct 30 20:36:10.670361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.669929    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" event={"ID":"f1022e3f-ea50-4bca-961f-e8a71e390086","Type":"ContainerDied","Data":"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"}
Oct 30 20:36:10.670361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.669978    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9" event={"ID":"f1022e3f-ea50-4bca-961f-e8a71e390086","Type":"ContainerDied","Data":"ce995be8755231bc61bf6eb138b272ac7140f49dbbbaa6b19bcc674862583ac0"}
Oct 30 20:36:10.670361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.670005    5085 scope.go:122] "RemoveContainer" containerID="08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"
Oct 30 20:36:10.670361 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.670155    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"
Oct 30 20:36:10.688504 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.688472    5085 scope.go:122] "RemoveContainer" containerID="08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"
Oct 30 20:36:10.689078 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:36:10.689046    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978\": container with ID starting with 08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978 not found: ID does not exist" containerID="08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"
Oct 30 20:36:10.689215 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.689086    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978"} err="failed to get container status \"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978\": rpc error: code = NotFound desc = could not find container \"08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978\": container with ID starting with 08a9ef2c3507cf832122aaf6b19548083d2faaeb9c82f8b0a45a4a856b1ce978 not found: ID does not exist"
Oct 30 20:36:10.869281 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.869248    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:36:10.876616 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:10.876588    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-b37684-757d546cc7-mrmt9"]
Oct 30 20:36:12.086105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:12.086072    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 20:36:12.631668 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:12.631637    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1022e3f-ea50-4bca-961f-e8a71e390086" path="/var/lib/kubelet/pods/f1022e3f-ea50-4bca-961f-e8a71e390086/volumes"
Oct 30 20:36:17.624008 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:17.623969    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:36:21.939731 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:36:21.939696    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:36:27.106523 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.106482    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main" containerID="cri-o://c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7" gracePeriod=45
Oct 30 20:36:27.386017 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.385988    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 20:36:27.555520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555490    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.555739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555545    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\" (UniqueName: \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.555739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555620    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.555739 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555693    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.555916 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555799    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:27.555916 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555870    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.556034 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.555946    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\") pod \"c8fc380b-ae22-4c61-a47e-5220c0baef93\" (UID: \"c8fc380b-ae22-4c61-a47e-5220c0baef93\") "
Oct 30 20:36:27.556126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.556102    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:27.556181 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.556149    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:27.556352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.556333    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.556413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.556355    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.556413 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.556369    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.558302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.558271    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:36:27.558302 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.558281    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "kube-api-access-2qbz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:36:27.558515 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.558361    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs" pod "c8fc380b-ae22-4c61-a47e-5220c0baef93" (UID: "c8fc380b-ae22-4c61-a47e-5220c0baef93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:36:27.657849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.657768    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qbz6\" (UniqueName: \"kubernetes.io/projected/c8fc380b-ae22-4c61-a47e-5220c0baef93-kube-api-access-2qbz6\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.657849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.657801    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8fc380b-ae22-4c61-a47e-5220c0baef93-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.657849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.657810    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8fc380b-ae22-4c61-a47e-5220c0baef93-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:36:27.783956 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.783915    5085 generic.go:356] "Generic (PLEG): container finished" podID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerID="c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7" exitCode=0
Oct 30 20:36:27.784130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.783961    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"
Oct 30 20:36:27.784130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.783997    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" event={"ID":"c8fc380b-ae22-4c61-a47e-5220c0baef93","Type":"ContainerDied","Data":"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"}
Oct 30 20:36:27.784130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.784038    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr" event={"ID":"c8fc380b-ae22-4c61-a47e-5220c0baef93","Type":"ContainerDied","Data":"0ab5581e46dae76c4355d43d6970dfbc1a12c80e5b1cc95b4e94d2daf6579ff7"}
Oct 30 20:36:27.784130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.784058    5085 scope.go:122] "RemoveContainer" containerID="c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"
Oct 30 20:36:27.794676 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.794657    5085 scope.go:122] "RemoveContainer" containerID="c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"
Oct 30 20:36:27.795021 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:36:27.794999    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7\": container with ID starting with c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7 not found: ID does not exist" containerID="c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"
Oct 30 20:36:27.795084 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.795030    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7"} err="failed to get container status \"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7\": rpc error: code = NotFound desc = could not find container \"c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7\": container with ID starting with c6d72247c5015978d7057d2c3e9765036dea1e7729c0efc955d05a943d031ce7 not found: ID does not exist"
Oct 30 20:36:27.954365 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.954335    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 20:36:27.958900 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:27.958876    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bbfa07/simulator-w3-bbfa07-kserve-86cc7c9458-r7mlr"]
Oct 30 20:36:28.636235 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:28.633599    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" path="/var/lib/kubelet/pods/c8fc380b-ae22-4c61-a47e-5220c0baef93/volumes"
Oct 30 20:36:32.195787 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:32.195746    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:36:51.941929 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:36:51.941892    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:36:59.376248 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:36:59.376212    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:37:10.526194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:10.526159    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:37:10.636044 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:10.636015    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:37:15.527103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:15.527032    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing" containerID="cri-o://1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0" gracePeriod=25
Oct 30 20:37:15.636757 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:15.636721    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing" containerID="cri-o://d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264" gracePeriod=25
Oct 30 20:37:19.508696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:19.508651    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:37:19.508696 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:19.508670    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:37:21.940237 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:37:21.940205    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:37:29.509123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:29.509032    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:37:29.509123 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:29.509078    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:37:35.092978 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.092945    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:37:35.802077 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.802005    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:37:35.890435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.890411    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:37:35.911370 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.911349    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\") pod \"95422b84-7885-4e95-b7c9-88dd2344bfea\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") "
Oct 30 20:37:35.911479 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.911389    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\" (UniqueName: \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\") pod \"95422b84-7885-4e95-b7c9-88dd2344bfea\" (UID: \"95422b84-7885-4e95-b7c9-88dd2344bfea\") "
Oct 30 20:37:35.911756 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.911732    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume" pod "95422b84-7885-4e95-b7c9-88dd2344bfea" (UID: "95422b84-7885-4e95-b7c9-88dd2344bfea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:35.913471 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:35.913451    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8" pod "95422b84-7885-4e95-b7c9-88dd2344bfea" (UID: "95422b84-7885-4e95-b7c9-88dd2344bfea"). InnerVolumeSpecName "kube-api-access-ptmt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:36.012792 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.012764    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\") pod \"bad9cec6-091a-48a4-afbf-2c86542c0df2\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") "
Oct 30 20:37:36.012962 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.012898    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\" (UniqueName: \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\") pod \"bad9cec6-091a-48a4-afbf-2c86542c0df2\" (UID: \"bad9cec6-091a-48a4-afbf-2c86542c0df2\") "
Oct 30 20:37:36.013160 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.013137    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume" pod "bad9cec6-091a-48a4-afbf-2c86542c0df2" (UID: "bad9cec6-091a-48a4-afbf-2c86542c0df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:36.013264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.013248    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bad9cec6-091a-48a4-afbf-2c86542c0df2-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:36.013329 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.013265    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95422b84-7885-4e95-b7c9-88dd2344bfea-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:36.013329 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.013274    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmt8\" (UniqueName: \"kubernetes.io/projected/95422b84-7885-4e95-b7c9-88dd2344bfea-kube-api-access-ptmt8\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:36.014996 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.014976    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7" pod "bad9cec6-091a-48a4-afbf-2c86542c0df2" (UID: "bad9cec6-091a-48a4-afbf-2c86542c0df2"). InnerVolumeSpecName "kube-api-access-sh4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:36.114288 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.114210    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4x7\" (UniqueName: \"kubernetes.io/projected/bad9cec6-091a-48a4-afbf-2c86542c0df2-kube-api-access-sh4x7\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:36.226217 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.226183    5085 generic.go:356] "Generic (PLEG): container finished" podID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerID="d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264" exitCode=0
Oct 30 20:37:36.226419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.226230    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"
Oct 30 20:37:36.226419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.226262    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" event={"ID":"bad9cec6-091a-48a4-afbf-2c86542c0df2","Type":"ContainerDied","Data":"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"}
Oct 30 20:37:36.226419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.226336    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp" event={"ID":"bad9cec6-091a-48a4-afbf-2c86542c0df2","Type":"ContainerDied","Data":"8499021afb44710fe9d6e45d91f4930ee40031d9a06d254e3bc432d1fed084d6"}
Oct 30 20:37:36.226419 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.226357    5085 scope.go:122] "RemoveContainer" containerID="d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"
Oct 30 20:37:36.227888 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.227861    5085 generic.go:356] "Generic (PLEG): container finished" podID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerID="1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0" exitCode=0
Oct 30 20:37:36.228021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.227902    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"
Oct 30 20:37:36.228021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.227916    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" event={"ID":"95422b84-7885-4e95-b7c9-88dd2344bfea","Type":"ContainerDied","Data":"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"}
Oct 30 20:37:36.228021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.227956    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2" event={"ID":"95422b84-7885-4e95-b7c9-88dd2344bfea","Type":"ContainerDied","Data":"0765c3932fa1e113b6465fa03e03aa55cd578e0d6fcb35c7d976cf068b6db12a"}
Oct 30 20:37:36.237454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.237438    5085 scope.go:122] "RemoveContainer" containerID="d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"
Oct 30 20:37:36.237735 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:37:36.237712    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264\": container with ID starting with d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264 not found: ID does not exist" containerID="d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"
Oct 30 20:37:36.237828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.237740    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264"} err="failed to get container status \"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264\": rpc error: code = NotFound desc = could not find container \"d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264\": container with ID starting with d6c1ca46309ce0294622d8d7765399843302de5ac10443d0c89ab494ecc9d264 not found: ID does not exist"
Oct 30 20:37:36.237828 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.237766    5085 scope.go:122] "RemoveContainer" containerID="1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"
Oct 30 20:37:36.246499 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.246481    5085 scope.go:122] "RemoveContainer" containerID="1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"
Oct 30 20:37:36.246774 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:37:36.246754    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0\": container with ID starting with 1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0 not found: ID does not exist" containerID="1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"
Oct 30 20:37:36.246824 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.246782    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0"} err="failed to get container status \"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0\": rpc error: code = NotFound desc = could not find container \"1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0\": container with ID starting with 1f92bc199da36c7beba1911c2eba11f4e907f38cd4d12bf022b7a688fe6909a0 not found: ID does not exist"
Oct 30 20:37:36.395493 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.395419    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:37:36.401150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.401118    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-547082-7f5584658c-gcnvp"]
Oct 30 20:37:36.523390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.523361    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:37:36.532382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.532357    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-547082-74fc5b6748-rnbv2"]
Oct 30 20:37:36.631694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.631662    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" path="/var/lib/kubelet/pods/95422b84-7885-4e95-b7c9-88dd2344bfea/volumes"
Oct 30 20:37:36.632126 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:36.632104    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" path="/var/lib/kubelet/pods/bad9cec6-091a-48a4-afbf-2c86542c0df2/volumes"
Oct 30 20:37:51.847578 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:51.847529    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:37:51.940380 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:37:51.940343    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:37:52.352420 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.352378    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:37:52.352773 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.352734    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" podUID="326632a3-36af-4fee-9a49-f32ad2114b56" containerName="maas-api" containerID="cri-o://90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495" gracePeriod=30
Oct 30 20:37:52.631111 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.631082    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:37:52.688984 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.688955    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\") pod \"326632a3-36af-4fee-9a49-f32ad2114b56\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") "
Oct 30 20:37:52.689152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.689002    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") pod \"326632a3-36af-4fee-9a49-f32ad2114b56\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") "
Oct 30 20:37:52.689152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.689024    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\" (UniqueName: \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\") pod \"326632a3-36af-4fee-9a49-f32ad2114b56\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") "
Oct 30 20:37:52.689293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.689178    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\") pod \"326632a3-36af-4fee-9a49-f32ad2114b56\" (UID: \"326632a3-36af-4fee-9a49-f32ad2114b56\") "
Oct 30 20:37:52.689627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.689597    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca" pod "326632a3-36af-4fee-9a49-f32ad2114b56" (UID: "326632a3-36af-4fee-9a49-f32ad2114b56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:37:52.689777 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.689733    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/326632a3-36af-4fee-9a49-f32ad2114b56-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:52.691253 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.691223    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls" pod "326632a3-36af-4fee-9a49-f32ad2114b56" (UID: "326632a3-36af-4fee-9a49-f32ad2114b56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:37:52.691358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.691248    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs" pod "326632a3-36af-4fee-9a49-f32ad2114b56" (UID: "326632a3-36af-4fee-9a49-f32ad2114b56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:37:52.691358 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.691292    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x" pod "326632a3-36af-4fee-9a49-f32ad2114b56" (UID: "326632a3-36af-4fee-9a49-f32ad2114b56"). InnerVolumeSpecName "kube-api-access-2pt2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:37:52.790727 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.790690    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:52.790727 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.790722    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/326632a3-36af-4fee-9a49-f32ad2114b56-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:52.790947 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:52.790735    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt2x\" (UniqueName: \"kubernetes.io/projected/326632a3-36af-4fee-9a49-f32ad2114b56-kube-api-access-2pt2x\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:37:53.336810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.336773    5085 generic.go:356] "Generic (PLEG): container finished" podID="326632a3-36af-4fee-9a49-f32ad2114b56" containerID="90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495" exitCode=0
Oct 30 20:37:53.337294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.336815    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" event={"ID":"326632a3-36af-4fee-9a49-f32ad2114b56","Type":"ContainerDied","Data":"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"}
Oct 30 20:37:53.337294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.336827    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"
Oct 30 20:37:53.337294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.336866    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9" event={"ID":"326632a3-36af-4fee-9a49-f32ad2114b56","Type":"ContainerDied","Data":"684d490cbec1b403a03094340181bbadcb7c0701d2fefa85151c643a616ad667"}
Oct 30 20:37:53.337294 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.336895    5085 scope.go:122] "RemoveContainer" containerID="90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"
Oct 30 20:37:53.347742 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.347726    5085 scope.go:122] "RemoveContainer" containerID="90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"
Oct 30 20:37:53.347990 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:37:53.347969    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495\": container with ID starting with 90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495 not found: ID does not exist" containerID="90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"
Oct 30 20:37:53.348030 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.348000    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495"} err="failed to get container status \"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495\": rpc error: code = NotFound desc = could not find container \"90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495\": container with ID starting with 90e726c00877689ba35e23fd975f168c184f0de7401beb6bd587413245862495 not found: ID does not exist"
Oct 30 20:37:53.543503 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.543467    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:37:53.594389 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:53.594325    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-547082-cb54557f9-tjjs9"]
Oct 30 20:37:54.631436 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:37:54.631395    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326632a3-36af-4fee-9a49-f32ad2114b56" path="/var/lib/kubelet/pods/326632a3-36af-4fee-9a49-f32ad2114b56/volumes"
Oct 30 20:38:00.435741 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:00.435705    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:38:00.436198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:00.435946    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy" containerID="cri-o://f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4" gracePeriod=30
Oct 30 20:38:05.702171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.702141    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:38:05.838586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838488    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838541    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838609    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838626    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838710    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838756    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.838806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838794    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.839179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838821    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.839179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838905    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\") pod \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\" (UID: \"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017\") "
Oct 30 20:38:05.839179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.838964    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:05.839179 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839121    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:05.839408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839218    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:05.839408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839240    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:05.839408 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839300    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:05.839592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839488    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.839592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839506    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.839592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839520    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.839592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839535    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.839592 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.839549    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.841105 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.841080    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:05.841228 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.841184    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:38:05.841272 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.841258    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "kube-api-access-9qxbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:05.841393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.841371    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy" pod "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" (UID: "aea6b2e4-fbe6-47a8-a87d-e97bd8a39017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:38:05.940415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.940380    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.940415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.940410    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxbl\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-kube-api-access-9qxbl\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.940415 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.940420    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:05.940671 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:05.940428    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:06.425433 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.425396    5085 generic.go:356] "Generic (PLEG): container finished" podID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerID="f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4" exitCode=0
Oct 30 20:38:06.425635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.425459    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"
Oct 30 20:38:06.425635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.425475    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" event={"ID":"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017","Type":"ContainerDied","Data":"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"}
Oct 30 20:38:06.425635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.425528    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx" event={"ID":"aea6b2e4-fbe6-47a8-a87d-e97bd8a39017","Type":"ContainerDied","Data":"efe9e9366bc67f844a3094ed177051c6eab0ab8683f83ee419611be388aedbc4"}
Oct 30 20:38:06.425635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.425550    5085 scope.go:122] "RemoveContainer" containerID="f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"
Oct 30 20:38:06.436647 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.436629    5085 scope.go:122] "RemoveContainer" containerID="f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"
Oct 30 20:38:06.436952 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:38:06.436931    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4\": container with ID starting with f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4 not found: ID does not exist" containerID="f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"
Oct 30 20:38:06.437023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.436961    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4"} err="failed to get container status \"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4\": rpc error: code = NotFound desc = could not find container \"f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4\": container with ID starting with f61a8ba5bf884595254fd0ffd9873ba58a4bcc4d0bc02ecd156b73cf694422b4 not found: ID does not exist"
Oct 30 20:38:06.602138 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.602103    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:38:06.652995 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:06.652960    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-98a19b-openshift-default-775879c667-j22qx"]
Oct 30 20:38:08.631542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:08.631497    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" path="/var/lib/kubelet/pods/aea6b2e4-fbe6-47a8-a87d-e97bd8a39017/volumes"
Oct 30 20:38:11.209965 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:11.209924    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:38:21.938248 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:38:21.938201    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:38:35.190061 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:35.190024    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:38:51.156884 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.156849    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:38:51.157345 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.157075    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" podUID="7ec237de-6db4-424b-a4bb-29516c034d7a" containerName="maas-api" containerID="cri-o://3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4" gracePeriod=30
Oct 30 20:38:51.447602 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.447550    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:38:51.517585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.517529    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\") pod \"7ec237de-6db4-424b-a4bb-29516c034d7a\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") "
Oct 30 20:38:51.517794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.517615    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\" (UniqueName: \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\") pod \"7ec237de-6db4-424b-a4bb-29516c034d7a\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") "
Oct 30 20:38:51.517794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.517687    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\") pod \"7ec237de-6db4-424b-a4bb-29516c034d7a\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") "
Oct 30 20:38:51.517794 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.517752    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\") pod \"7ec237de-6db4-424b-a4bb-29516c034d7a\" (UID: \"7ec237de-6db4-424b-a4bb-29516c034d7a\") "
Oct 30 20:38:51.518003 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.517897    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca" pod "7ec237de-6db4-424b-a4bb-29516c034d7a" (UID: "7ec237de-6db4-424b-a4bb-29516c034d7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:38:51.518081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.518050    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ec237de-6db4-424b-a4bb-29516c034d7a-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:51.520518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.520486    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls" pod "7ec237de-6db4-424b-a4bb-29516c034d7a" (UID: "7ec237de-6db4-424b-a4bb-29516c034d7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:51.520799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.520770    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs" pod "7ec237de-6db4-424b-a4bb-29516c034d7a" (UID: "7ec237de-6db4-424b-a4bb-29516c034d7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:38:51.520955 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.520896    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5" pod "7ec237de-6db4-424b-a4bb-29516c034d7a" (UID: "7ec237de-6db4-424b-a4bb-29516c034d7a"). InnerVolumeSpecName "kube-api-access-tvhg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:38:51.619502 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.619467    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvhg5\" (UniqueName: \"kubernetes.io/projected/7ec237de-6db4-424b-a4bb-29516c034d7a-kube-api-access-tvhg5\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:51.619502 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.619497    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:51.619502 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.619508    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ec237de-6db4-424b-a4bb-29516c034d7a-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:38:51.701347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.701260    5085 generic.go:356] "Generic (PLEG): container finished" podID="7ec237de-6db4-424b-a4bb-29516c034d7a" containerID="3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4" exitCode=0
Oct 30 20:38:51.701347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.701316    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"
Oct 30 20:38:51.701347 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.701335    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" event={"ID":"7ec237de-6db4-424b-a4bb-29516c034d7a","Type":"ContainerDied","Data":"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"}
Oct 30 20:38:51.701715 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.701387    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht" event={"ID":"7ec237de-6db4-424b-a4bb-29516c034d7a","Type":"ContainerDied","Data":"0e77dfee208990c866e58ec3e35ae905c87e945bb5716e16078e9d40abcb8b8c"}
Oct 30 20:38:51.701715 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.701415    5085 scope.go:122] "RemoveContainer" containerID="3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"
Oct 30 20:38:51.712741 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.712721    5085 scope.go:122] "RemoveContainer" containerID="3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"
Oct 30 20:38:51.713044 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:38:51.713022    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4\": container with ID starting with 3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4 not found: ID does not exist" containerID="3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"
Oct 30 20:38:51.713113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.713058    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4"} err="failed to get container status \"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4\": rpc error: code = NotFound desc = could not find container \"3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4\": container with ID starting with 3a7abc25b5a07e2800cd61fe3693644471c94e17448475dd1978229f62b36df4 not found: ID does not exist"
Oct 30 20:38:51.884285 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.884245    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:38:51.889635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:51.889609    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e4864d-c4b8ddc85-xccht"]
Oct 30 20:38:51.939052 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:38:51.939020    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:38:52.631129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:38:52.631094    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec237de-6db4-424b-a4bb-29516c034d7a" path="/var/lib/kubelet/pods/7ec237de-6db4-424b-a4bb-29516c034d7a/volumes"
Oct 30 20:39:01.653452 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:01.653363    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:39:06.089620 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:06.089580    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 20:39:21.110004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.109957    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main" containerID="cri-o://0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd" gracePeriod=45
Oct 30 20:39:21.374072 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.374009    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 20:39:21.437079 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437048    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437104    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437155    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437209    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437282 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437265    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\" (UniqueName: \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437351    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\" (UniqueName: \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\") pod \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\" (UID: \"291fdcd5-7ff1-4eca-be25-dfb1423d126b\") "
Oct 30 20:39:21.437835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.437804    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:21.438260 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.438220    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:21.438375 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.438343    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:21.439863 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.439841    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:39:21.440133 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.440103    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:39:21.440490 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.440467    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw" pod "291fdcd5-7ff1-4eca-be25-dfb1423d126b" (UID: "291fdcd5-7ff1-4eca-be25-dfb1423d126b"). InnerVolumeSpecName "kube-api-access-7m9bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:39:21.538799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538768    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.538799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538794    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.538799 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538804    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.539156 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538812    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.539156 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538820    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/291fdcd5-7ff1-4eca-be25-dfb1423d126b-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.539156 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.538838    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m9bw\" (UniqueName: \"kubernetes.io/projected/291fdcd5-7ff1-4eca-be25-dfb1423d126b-kube-api-access-7m9bw\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:39:21.887292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.881540    5085 generic.go:356] "Generic (PLEG): container finished" podID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerID="0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd" exitCode=0
Oct 30 20:39:21.887292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.881596    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" event={"ID":"291fdcd5-7ff1-4eca-be25-dfb1423d126b","Type":"ContainerDied","Data":"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"}
Oct 30 20:39:21.887292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.881615    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"
Oct 30 20:39:21.887292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.881642    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc" event={"ID":"291fdcd5-7ff1-4eca-be25-dfb1423d126b","Type":"ContainerDied","Data":"123157c30a0a78b6e899379f8cd2fb6f3615af35223a278587a55a908c6962a7"}
Oct 30 20:39:21.887292 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.881663    5085 scope.go:122] "RemoveContainer" containerID="0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"
Oct 30 20:39:21.894752 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.894722    5085 scope.go:122] "RemoveContainer" containerID="0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"
Oct 30 20:39:21.895169 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:39:21.895131    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd\": container with ID starting with 0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd not found: ID does not exist" containerID="0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"
Oct 30 20:39:21.895286 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:21.895176    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd"} err="failed to get container status \"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd\": rpc error: code = NotFound desc = could not find container \"0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd\": container with ID starting with 0ec9ae482df05e3a18703af3f7bb41449ad61fc7395219128fef861c56fc7bbd not found: ID does not exist"
Oct 30 20:39:21.939344 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:39:21.939310    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:39:22.065817 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:22.065779    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 20:39:22.068368 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:22.068346    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a74d15/simulator-w0-a74d15-kserve-7fd84895c7-rbtrc"]
Oct 30 20:39:22.631129 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:22.631094    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" path="/var/lib/kubelet/pods/291fdcd5-7ff1-4eca-be25-dfb1423d126b/volumes"
Oct 30 20:39:49.872704 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:49.872667    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:39:51.629243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:51.629212    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:39:51.642907 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:39:51.642882    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-gc2t4_b0daabe8-7547-4768-9157-55e9a70f1fc1/console-operator/1.log"
Oct 30 20:39:51.939581 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:39:51.939533    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:40:09.992078 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:09.992044    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 20:40:09.992548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:09.992272    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy" containerID="cri-o://7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45" gracePeriod=30
Oct 30 20:40:15.262545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.262479    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 20:40:15.309432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.309401    5085 generic.go:356] "Generic (PLEG): container finished" podID="c75cb793-10a5-431b-9086-130755cb8381" containerID="7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45" exitCode=0
Oct 30 20:40:15.309622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.309456    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"
Oct 30 20:40:15.309622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.309480    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" event={"ID":"c75cb793-10a5-431b-9086-130755cb8381","Type":"ContainerDied","Data":"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"}
Oct 30 20:40:15.309622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.309575    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx" event={"ID":"c75cb793-10a5-431b-9086-130755cb8381","Type":"ContainerDied","Data":"3839bbda6208b56f71bd076d12811eaeddf272ab00262c3f709a0668f4413e52"}
Oct 30 20:40:15.309622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.309598    5085 scope.go:122] "RemoveContainer" containerID="7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"
Oct 30 20:40:15.319542 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.319521    5085 scope.go:122] "RemoveContainer" containerID="7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"
Oct 30 20:40:15.319844 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:40:15.319824    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45\": container with ID starting with 7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45 not found: ID does not exist" containerID="7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"
Oct 30 20:40:15.319927 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.319853    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45"} err="failed to get container status \"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45\": rpc error: code = NotFound desc = could not find container \"7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45\": container with ID starting with 7df34f54e00981c40ec56baf2fdaefc11e93f4dd030608dd2ec06889e255bb45 not found: ID does not exist"
Oct 30 20:40:15.427470 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427438    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427717 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427533    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427717 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427632    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427717 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427682    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427720    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427797    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.427898 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427881    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.428064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427932    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.428064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427960    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\") pod \"c75cb793-10a5-431b-9086-130755cb8381\" (UID: \"c75cb793-10a5-431b-9086-130755cb8381\") "
Oct 30 20:40:15.428064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427969    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:15.428064 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427964    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:15.428264 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.428196    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:15.428323 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.427949    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:15.428483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.428371    5085 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.428483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.428420    5085 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c75cb793-10a5-431b-9086-130755cb8381-istiod-ca-cert\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.428483 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.428459    5085 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-credential-socket\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.428767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.428486    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:15.430190 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.430155    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:40:15.430319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.430285    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "kube-api-access-krgxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:15.430319 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.430297    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:15.430422 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.430397    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo" pod "c75cb793-10a5-431b-9086-130755cb8381" (UID: "c75cb793-10a5-431b-9086-130755cb8381"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:40:15.529693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529604    5085 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-data\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.529693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529642    5085 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c75cb793-10a5-431b-9086-130755cb8381-istio-podinfo\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.529693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529658    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krgxg\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-kube-api-access-krgxg\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.529693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529675    5085 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-istio-envoy\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.529693 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529689    5085 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c75cb793-10a5-431b-9086-130755cb8381-workload-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.529997 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.529703    5085 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c75cb793-10a5-431b-9086-130755cb8381-istio-token\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:15.734505 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.734474    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 20:40:15.740244 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:15.740218    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a74d15-openshift-default-6659bdd657-ml9rx"]
Oct 30 20:40:16.631645 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:16.631613    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75cb793-10a5-431b-9086-130755cb8381" path="/var/lib/kubelet/pods/c75cb793-10a5-431b-9086-130755cb8381/volumes"
Oct 30 20:40:17.331525 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:17.331491    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:40:21.939162 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:40:21.939123    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:40:22.249230 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.249144    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:40:22.249412 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.249351    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" podUID="002593c7-ca36-4d55-9cbf-270d2417426c" containerName="maas-api" containerID="cri-o://30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f" gracePeriod=30
Oct 30 20:40:22.517337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.517269    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:40:22.610977 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.610939    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\" (UniqueName: \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\") pod \"002593c7-ca36-4d55-9cbf-270d2417426c\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") "
Oct 30 20:40:22.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.611036    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") pod \"002593c7-ca36-4d55-9cbf-270d2417426c\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") "
Oct 30 20:40:22.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.611081    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\") pod \"002593c7-ca36-4d55-9cbf-270d2417426c\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") "
Oct 30 20:40:22.611139 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.611112    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\") pod \"002593c7-ca36-4d55-9cbf-270d2417426c\" (UID: \"002593c7-ca36-4d55-9cbf-270d2417426c\") "
Oct 30 20:40:22.611442 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.611420    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca" pod "002593c7-ca36-4d55-9cbf-270d2417426c" (UID: "002593c7-ca36-4d55-9cbf-270d2417426c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:40:22.611545 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.611527    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/002593c7-ca36-4d55-9cbf-270d2417426c-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:22.613268 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.613242    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km" pod "002593c7-ca36-4d55-9cbf-270d2417426c" (UID: "002593c7-ca36-4d55-9cbf-270d2417426c"). InnerVolumeSpecName "kube-api-access-vv7km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:40:22.613373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.613278    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs" pod "002593c7-ca36-4d55-9cbf-270d2417426c" (UID: "002593c7-ca36-4d55-9cbf-270d2417426c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:22.613373 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.613299    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls" pod "002593c7-ca36-4d55-9cbf-270d2417426c" (UID: "002593c7-ca36-4d55-9cbf-270d2417426c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:40:22.712065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.712030    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:22.712065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.712059    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/002593c7-ca36-4d55-9cbf-270d2417426c-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:22.712065 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:22.712070    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv7km\" (UniqueName: \"kubernetes.io/projected/002593c7-ca36-4d55-9cbf-270d2417426c-kube-api-access-vv7km\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:40:23.360188 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.360145    5085 generic.go:356] "Generic (PLEG): container finished" podID="002593c7-ca36-4d55-9cbf-270d2417426c" containerID="30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f" exitCode=0
Oct 30 20:40:23.360760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.360197    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"
Oct 30 20:40:23.360760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.360225    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" event={"ID":"002593c7-ca36-4d55-9cbf-270d2417426c","Type":"ContainerDied","Data":"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"}
Oct 30 20:40:23.360760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.360292    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92" event={"ID":"002593c7-ca36-4d55-9cbf-270d2417426c","Type":"ContainerDied","Data":"9b45a7a9604254275ec01da2d869de7043e0826d21cfe765e4a46b2769738b0c"}
Oct 30 20:40:23.360760 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.360346    5085 scope.go:122] "RemoveContainer" containerID="30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"
Oct 30 20:40:23.370508 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.370483    5085 scope.go:122] "RemoveContainer" containerID="30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"
Oct 30 20:40:23.370842 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:40:23.370809    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f\": container with ID starting with 30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f not found: ID does not exist" containerID="30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"
Oct 30 20:40:23.370895 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.370851    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f"} err="failed to get container status \"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f\": rpc error: code = NotFound desc = could not find container \"30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f\": container with ID starting with 30370f2e4722dc2ca8308ab1b0e3a00d7b946c32bbe1cc4d1777e7f3cbec877f not found: ID does not exist"
Oct 30 20:40:23.547874 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.547838    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:40:23.598615 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:23.598579    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-46dadd-67f8bc6fcb-ggb92"]
Oct 30 20:40:24.630694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:24.630660    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002593c7-ca36-4d55-9cbf-270d2417426c" path="/var/lib/kubelet/pods/002593c7-ca36-4d55-9cbf-270d2417426c/volumes"
Oct 30 20:40:35.729249 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:35.729166    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:40:51.938549 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:40:51.938513    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:40:55.926348 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:40:55.926299    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:41:21.856887 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:21.856852    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:41:21.857301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:21.857082    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" podUID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" containerName="maas-api" containerID="cri-o://d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e" gracePeriod=30
Oct 30 20:41:21.941995 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:21.941959    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:41:22.122881 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.122785    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:41:22.223364 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223333    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\") pod \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") "
Oct 30 20:41:22.223547 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223481    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\" (UniqueName: \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\") pod \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") "
Oct 30 20:41:22.223655 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223543    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\") pod \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") "
Oct 30 20:41:22.223655 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223610    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") pod \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\" (UID: \"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41\") "
Oct 30 20:41:22.223801 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223776    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca" pod "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" (UID: "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:22.223988 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.223967    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:22.225694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.225662    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm" pod "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" (UID: "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41"). InnerVolumeSpecName "kube-api-access-6c4bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:22.225694 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.225664    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs" pod "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" (UID: "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:22.225838 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.225720    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls" pod "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" (UID: "2fac75e5-dcc7-4feb-b6be-3e37a13a6e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:22.325204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.325166    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4bm\" (UniqueName: \"kubernetes.io/projected/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-kube-api-access-6c4bm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:22.325204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.325200    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:22.325204 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.325211    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:22.701414 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.701369    5085 generic.go:356] "Generic (PLEG): container finished" podID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" containerID="d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e" exitCode=0
Oct 30 20:41:22.701628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.701415    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"
Oct 30 20:41:22.701628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.701436    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" event={"ID":"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41","Type":"ContainerDied","Data":"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"}
Oct 30 20:41:22.701628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.701495    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q" event={"ID":"2fac75e5-dcc7-4feb-b6be-3e37a13a6e41","Type":"ContainerDied","Data":"ce2a239d56260cacb9eac02839f1cb15951d59a8c78cb81296fdbbf2c0a13ffa"}
Oct 30 20:41:22.701628 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.701521    5085 scope.go:122] "RemoveContainer" containerID="d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"
Oct 30 20:41:22.711141 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.711116    5085 scope.go:122] "RemoveContainer" containerID="d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"
Oct 30 20:41:22.711395 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:22.711373    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e\": container with ID starting with d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e not found: ID does not exist" containerID="d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"
Oct 30 20:41:22.711444 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.711406    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e"} err="failed to get container status \"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e\": rpc error: code = NotFound desc = could not find container \"d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e\": container with ID starting with d36f3fe870c2a401dba518522406813f7beda4ec34812c2fd28d2a631693764e not found: ID does not exist"
Oct 30 20:41:22.871665 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.871633    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:41:22.891103 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:22.891023    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-056a27-bc9b84bf6-ls96q"]
Oct 30 20:41:24.630585 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:24.630526    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" path="/var/lib/kubelet/pods/2fac75e5-dcc7-4feb-b6be-3e37a13a6e41/volumes"
Oct 30 20:41:27.704081 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:27.704049    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:41:44.239983 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:44.239949    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:41:44.347654 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:44.347621    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:41:49.240537 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:49.240476    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing" containerID="cri-o://880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e" gracePeriod=25
Oct 30 20:41:49.348481 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:49.348441    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing" containerID="cri-o://c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4" gracePeriod=25
Oct 30 20:41:51.939209 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:51.939169    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:41:54.402134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.402105    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:41:54.402576 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.402304    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" podUID="94420e6d-c47e-4be9-9b91-513eaf86bc29" containerName="maas-api" containerID="cri-o://cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf" gracePeriod=30
Oct 30 20:41:54.653280 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.653216    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:41:54.727508 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.727476    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\") pod \"94420e6d-c47e-4be9-9b91-513eaf86bc29\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") "
Oct 30 20:41:54.727730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.727520    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") pod \"94420e6d-c47e-4be9-9b91-513eaf86bc29\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") "
Oct 30 20:41:54.727730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.727635    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\" (UniqueName: \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\") pod \"94420e6d-c47e-4be9-9b91-513eaf86bc29\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") "
Oct 30 20:41:54.727730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.727705    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\") pod \"94420e6d-c47e-4be9-9b91-513eaf86bc29\" (UID: \"94420e6d-c47e-4be9-9b91-513eaf86bc29\") "
Oct 30 20:41:54.727926 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.727909    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca" pod "94420e6d-c47e-4be9-9b91-513eaf86bc29" (UID: "94420e6d-c47e-4be9-9b91-513eaf86bc29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:41:54.728040 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.728019    5085 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94420e6d-c47e-4be9-9b91-513eaf86bc29-openshift-service-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:54.729738 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.729711    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x" pod "94420e6d-c47e-4be9-9b91-513eaf86bc29" (UID: "94420e6d-c47e-4be9-9b91-513eaf86bc29"). InnerVolumeSpecName "kube-api-access-mtv6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:41:54.729845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.729733    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs" pod "94420e6d-c47e-4be9-9b91-513eaf86bc29" (UID: "94420e6d-c47e-4be9-9b91-513eaf86bc29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:54.729845 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.729813    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls" pod "94420e6d-c47e-4be9-9b91-513eaf86bc29" (UID: "94420e6d-c47e-4be9-9b91-513eaf86bc29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:41:54.828548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.828515    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtv6x\" (UniqueName: \"kubernetes.io/projected/94420e6d-c47e-4be9-9b91-513eaf86bc29-kube-api-access-mtv6x\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:54.828548 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.828546    5085 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-metrics-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:54.828761 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.828556    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94420e6d-c47e-4be9-9b91-513eaf86bc29-maas-api-tls\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:41:54.886961 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.886927    5085 generic.go:356] "Generic (PLEG): container finished" podID="94420e6d-c47e-4be9-9b91-513eaf86bc29" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf" exitCode=0
Oct 30 20:41:54.887130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.886971    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"
Oct 30 20:41:54.887130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.887015    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" event={"ID":"94420e6d-c47e-4be9-9b91-513eaf86bc29","Type":"ContainerDied","Data":"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"}
Oct 30 20:41:54.887130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.887071    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc" event={"ID":"94420e6d-c47e-4be9-9b91-513eaf86bc29","Type":"ContainerDied","Data":"6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042"}
Oct 30 20:41:54.887130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.887098    5085 scope.go:122] "RemoveContainer" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:54.896305 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.896285    5085 scope.go:122] "RemoveContainer" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:54.898520 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.898500    5085 scope.go:122] "RemoveContainer" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:54.898787 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:54.898769    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf\": container with ID starting with cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf not found: ID does not exist" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:54.898842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:54.898794    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"} err="failed to get container status \"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf\": rpc error: code = NotFound desc = could not find container \"cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf\": container with ID starting with cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf not found: ID does not exist"
Oct 30 20:41:54.899366 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:54.899349    5085 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc_odh-ai-gateway-infra_94420e6d-c47e-4be9-9b91-513eaf86bc29_0 in pod sandbox 6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042 from index: no such id: 'cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf'" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:54.899412 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:41:54.899373    5085 kuberuntime_gc.go:151] "Failed to remove container" err="rpc error: code = Unknown desc = failed to delete container k8s_maas-api_maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc_odh-ai-gateway-infra_94420e6d-c47e-4be9-9b91-513eaf86bc29_0 in pod sandbox 6ea0e4ebf05121415b006c5a5aeb86b29de4c048bbcbea2fb4e33afa0c80a042 from index: no such id: 'cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf'" containerID="cac206703792811ff1e3cabb1ea8399b7285d2e0222d8924d43907fe052d97cf"
Oct 30 20:41:55.056835 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:55.056798    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:41:55.061021 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:55.060992    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-64120b-7f7fd859c9-rk5pc"]
Oct 30 20:41:56.630485 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:56.630447    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94420e6d-c47e-4be9-9b91-513eaf86bc29" path="/var/lib/kubelet/pods/94420e6d-c47e-4be9-9b91-513eaf86bc29/volumes"
Oct 30 20:41:56.945931 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:56.945893    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:41:57.953162 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:41:57.953079    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:42:00.222052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:00.222017    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:42:06.945922 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:06.945875    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:42:07.952886 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:07.952849    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:42:09.218293 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.218253    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 20:42:09.269301 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.269269    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 20:42:09.506868 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.506841    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:42:09.575581 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.575541    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\") pod \"c9738b32-9f30-4648-9f56-a691a8c41a67\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") "
Oct 30 20:42:09.575762 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.575734    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\" (UniqueName: \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\") pod \"c9738b32-9f30-4648-9f56-a691a8c41a67\" (UID: \"c9738b32-9f30-4648-9f56-a691a8c41a67\") "
Oct 30 20:42:09.575940 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.575915    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume" pod "c9738b32-9f30-4648-9f56-a691a8c41a67" (UID: "c9738b32-9f30-4648-9f56-a691a8c41a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:09.576087 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.576066    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9738b32-9f30-4648-9f56-a691a8c41a67-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:09.577784 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.577761    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv" pod "c9738b32-9f30-4648-9f56-a691a8c41a67" (UID: "c9738b32-9f30-4648-9f56-a691a8c41a67"). InnerVolumeSpecName "kube-api-access-fskkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:09.593735 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.593712    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:42:09.676468 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.676415    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\" (UniqueName: \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\") pod \"839c7660-f036-4c85-8f6e-767db2782d14\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") "
Oct 30 20:42:09.676688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.676484    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\") pod \"839c7660-f036-4c85-8f6e-767db2782d14\" (UID: \"839c7660-f036-4c85-8f6e-767db2782d14\") "
Oct 30 20:42:09.676920 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.676901    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fskkv\" (UniqueName: \"kubernetes.io/projected/c9738b32-9f30-4648-9f56-a691a8c41a67-kube-api-access-fskkv\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:09.676992 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.676900    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume" pod "839c7660-f036-4c85-8f6e-767db2782d14" (UID: "839c7660-f036-4c85-8f6e-767db2782d14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:09.678552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.678530    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr" pod "839c7660-f036-4c85-8f6e-767db2782d14" (UID: "839c7660-f036-4c85-8f6e-767db2782d14"). InnerVolumeSpecName "kube-api-access-4btvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:09.777934 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.777851    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4btvr\" (UniqueName: \"kubernetes.io/projected/839c7660-f036-4c85-8f6e-767db2782d14-kube-api-access-4btvr\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:09.777934 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.777877    5085 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839c7660-f036-4c85-8f6e-767db2782d14-plugins-config-volume\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:09.976002 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.975966    5085 generic.go:356] "Generic (PLEG): container finished" podID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerID="880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e" exitCode=0
Oct 30 20:42:09.976210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.976023    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"
Oct 30 20:42:09.976210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.976055    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" event={"ID":"c9738b32-9f30-4648-9f56-a691a8c41a67","Type":"ContainerDied","Data":"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"}
Oct 30 20:42:09.976210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.976114    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n" event={"ID":"c9738b32-9f30-4648-9f56-a691a8c41a67","Type":"ContainerDied","Data":"374ce953d4a9babb9146b73dae8e61ae2b67382dd66c230cf92e77f3d7c91f94"}
Oct 30 20:42:09.976210 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.976142    5085 scope.go:122] "RemoveContainer" containerID="880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"
Oct 30 20:42:09.977524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.977501    5085 generic.go:356] "Generic (PLEG): container finished" podID="839c7660-f036-4c85-8f6e-767db2782d14" containerID="c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4" exitCode=0
Oct 30 20:42:09.977687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.977535    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" event={"ID":"839c7660-f036-4c85-8f6e-767db2782d14","Type":"ContainerDied","Data":"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"}
Oct 30 20:42:09.977687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.977554    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"
Oct 30 20:42:09.977687 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.977594    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb" event={"ID":"839c7660-f036-4c85-8f6e-767db2782d14","Type":"ContainerDied","Data":"98fdd2ddc60b2d09adfdcada4b5053e5e8fdad6ec7d823b0d6fc198628f73132"}
Oct 30 20:42:09.988352 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.988332    5085 scope.go:122] "RemoveContainer" containerID="880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"
Oct 30 20:42:09.988652 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:09.988628    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e\": container with ID starting with 880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e not found: ID does not exist" containerID="880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"
Oct 30 20:42:09.988730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.988666    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e"} err="failed to get container status \"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e\": rpc error: code = NotFound desc = could not find container \"880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e\": container with ID starting with 880feafbe3d4179679e1675873db36357f8b0146efaa6ae27d913078c69b5e8e not found: ID does not exist"
Oct 30 20:42:09.988730 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.988689    5085 scope.go:122] "RemoveContainer" containerID="c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"
Oct 30 20:42:09.997193 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.997175    5085 scope.go:122] "RemoveContainer" containerID="c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"
Oct 30 20:42:09.997451 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:09.997432    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4\": container with ID starting with c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4 not found: ID does not exist" containerID="c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"
Oct 30 20:42:09.997514 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:09.997458    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4"} err="failed to get container status \"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4\": rpc error: code = NotFound desc = could not find container \"c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4\": container with ID starting with c2d44f81cd3ea1456284e72b57dbc56b9b1007a20728dec009b7c57d183cf9e4 not found: ID does not exist"
Oct 30 20:42:10.156043 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.155968    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:42:10.206609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.206578    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-64120b-59c4c64895-tbsbb"]
Oct 30 20:42:10.277728 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.277695    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:42:10.281187 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.281161    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-64120b-59d74fc496-ttj7n"]
Oct 30 20:42:10.630186 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.630157    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839c7660-f036-4c85-8f6e-767db2782d14" path="/var/lib/kubelet/pods/839c7660-f036-4c85-8f6e-767db2782d14/volumes"
Oct 30 20:42:10.630617 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:10.630601    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" path="/var/lib/kubelet/pods/c9738b32-9f30-4648-9f56-a691a8c41a67/volumes"
Oct 30 20:42:21.938188 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:21.938150    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:42:22.105864 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.105828    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"]
Oct 30 20:42:22.106427 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106410    5085 cpu_manager.go:414] "containerMap: removing container" podUID="326632a3-36af-4fee-9a49-f32ad2114b56" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106429    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326632a3-36af-4fee-9a49-f32ad2114b56" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106442    5085 cpu_manager.go:414] "containerMap: removing container" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106447    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106457    5085 cpu_manager.go:414] "containerMap: removing container" podUID="7ec237de-6db4-424b-a4bb-29516c034d7a" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106462    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec237de-6db4-424b-a4bb-29516c034d7a" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106473    5085 cpu_manager.go:414] "containerMap: removing container" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106478    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106485    5085 cpu_manager.go:414] "containerMap: removing container" podUID="94420e6d-c47e-4be9-9b91-513eaf86bc29" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106490    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94420e6d-c47e-4be9-9b91-513eaf86bc29" containerName="maas-api"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106502    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106507    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy"
Oct 30 20:42:22.106512 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106514    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106519    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106530    5085 cpu_manager.go:414] "containerMap: removing container" podUID="f1022e3f-ea50-4bca-961f-e8a71e390086" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106534    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1022e3f-ea50-4bca-961f-e8a71e390086" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106541    5085 cpu_manager.go:414] "containerMap: removing container" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106548    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106557    5085 cpu_manager.go:414] "containerMap: removing container" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106578    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106590    5085 cpu_manager.go:414] "containerMap: removing container" podUID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106595    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106600    5085 cpu_manager.go:414] "containerMap: removing container" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106604    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106611    5085 cpu_manager.go:414] "containerMap: removing container" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106616    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106623    5085 cpu_manager.go:414] "containerMap: removing container" podUID="002593c7-ca36-4d55-9cbf-270d2417426c" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106627    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002593c7-ca36-4d55-9cbf-270d2417426c" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106641    5085 cpu_manager.go:414] "containerMap: removing container" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106646    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106652    5085 cpu_manager.go:414] "containerMap: removing container" podUID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106656    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106664    5085 cpu_manager.go:414] "containerMap: removing container" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106668    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106678    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106682    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106692    5085 cpu_manager.go:414] "containerMap: removing container" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106696    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106774    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fc380b-ae22-4c61-a47e-5220c0baef93" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106784    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="002593c7-ca36-4d55-9cbf-270d2417426c" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106794    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="95422b84-7885-4e95-b7c9-88dd2344bfea" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106801    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fac75e5-dcc7-4feb-b6be-3e37a13a6e41" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106807    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea6b2e4-fbe6-47a8-a87d-e97bd8a39017" containerName="istio-proxy"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106815    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac207a5-b4f6-4897-b724-5533aa03dac2" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106820    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="291fdcd5-7ff1-4eca-be25-dfb1423d126b" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106834    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="326632a3-36af-4fee-9a49-f32ad2114b56" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106840    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="839c7660-f036-4c85-8f6e-767db2782d14" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106845    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec237de-6db4-424b-a4bb-29516c034d7a" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106853    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60f85e1-ae7b-4a9c-8f7b-65b740e6b545" containerName="main"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106860    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="94420e6d-c47e-4be9-9b91-513eaf86bc29" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106865    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75cb793-10a5-431b-9086-130755cb8381" containerName="istio-proxy"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106871    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9738b32-9f30-4648-9f56-a691a8c41a67" containerName="payload-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106877    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1022e3f-ea50-4bca-961f-e8a71e390086" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106881    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="bad9cec6-091a-48a4-afbf-2c86542c0df2" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106887    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fda8e3-64e1-4da7-b2dd-9cb95994254b" containerName="payload-pre-processing"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106893    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5b87b5-dec5-43f2-8d44-e595f3964ad1" containerName="maas-api"
Oct 30 20:42:22.106960 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.106897    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="af006b12-d6df-4191-b47a-7f3100f1416b" containerName="maas-api"
Oct 30 20:42:22.111849 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.111821    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.180612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.180575    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"]
Oct 30 20:42:22.228627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.228516    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:42:22.228627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.228529    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kmwqm\""
Oct 30 20:42:22.305521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.305486    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.305521 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.305523    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.305814 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.305637    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n8g9\" (UniqueName: \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.406580 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.406532    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.406786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.406606    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.406786 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.406665    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n8g9\" (UniqueName: \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.407240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.407214    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.408885 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.408865    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.469152 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.469120    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n8g9\" (UniqueName: \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\") pod \"maas-api-revoke-sub-890606b6-kzmx5\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:22.729404 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:22.729372    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:23.100148 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:42:23.100119    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644e9ab7_363f_420d_9d65_f66b5ab8164f.slice/crio-33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18 WatchSource:0}: Error finding container 33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18: Status 404 returned error can't find the container with id 33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18
Oct 30 20:42:23.102023 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:23.101935    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"]
Oct 30 20:42:24.058988 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.058946    5085 generic.go:356] "Generic (PLEG): container finished" podID="644e9ab7-363f-420d-9d65-f66b5ab8164f" containerID="ae837056725f636e18fbb6cc2ac7b25e6c0b5735991a7070ecfe52a4a54f9bac" exitCode=0
Oct 30 20:42:24.059169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.059009    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5" event={"ID":"644e9ab7-363f-420d-9d65-f66b5ab8164f","Type":"ContainerDied","Data":"ae837056725f636e18fbb6cc2ac7b25e6c0b5735991a7070ecfe52a4a54f9bac"}
Oct 30 20:42:24.059169 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.059058    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5" event={"ID":"644e9ab7-363f-420d-9d65-f66b5ab8164f","Type":"ContainerStarted","Data":"33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18"}
Oct 30 20:42:24.238645 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.238605    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main" containerID="cri-o://a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016" gracePeriod=45
Oct 30 20:42:24.289816 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.289782    5085 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" containerID="cri-o://24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9" gracePeriod=45
Oct 30 20:42:24.551166 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.551140    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 20:42:24.583627 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.583600    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 20:42:24.627902 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.627870    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.630516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.630490    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:24.729070 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729037    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729079    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729107    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729127    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729192    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\" (UniqueName: \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729255    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729318 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729285    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729337    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729362    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729394    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729442    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\") pod \"69ce92c4-2611-41d2-91da-1f76052c76d6\" (UID: \"69ce92c4-2611-41d2-91da-1f76052c76d6\") "
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729474    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\" (UniqueName: \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\") pod \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\" (UID: \"5dfd1e13-ef55-4072-9a50-468802aa4fc1\") "
Oct 30 20:42:24.729785 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.729735    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.730246 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730221    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.730477 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730431    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.730611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730513    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.730688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730639    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.730748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730687    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.730748 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730707    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.730906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.730879    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.732141 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.732103    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "kube-api-access-pxcxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:24.734230 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.734207    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:24.734331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.734255    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm" pod "69ce92c4-2611-41d2-91da-1f76052c76d6" (UID: "69ce92c4-2611-41d2-91da-1f76052c76d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.734331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.734267    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:42:24.734331 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.734261    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm" pod "5dfd1e13-ef55-4072-9a50-468802aa4fc1" (UID: "5dfd1e13-ef55-4072-9a50-468802aa4fc1"). InnerVolumeSpecName "kube-api-access-7gkqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831368    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxcxs\" (UniqueName: \"kubernetes.io/projected/69ce92c4-2611-41d2-91da-1f76052c76d6-kube-api-access-pxcxs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831399    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831410    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831418    5085 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-tmp-dir\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831428    5085 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69ce92c4-2611-41d2-91da-1f76052c76d6-tls-certs\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831436    5085 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-home\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831444    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gkqm\" (UniqueName: \"kubernetes.io/projected/5dfd1e13-ef55-4072-9a50-468802aa4fc1-kube-api-access-7gkqm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831452    5085 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69ce92c4-2611-41d2-91da-1f76052c76d6-dshm\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:24.831454 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:24.831460    5085 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dfd1e13-ef55-4072-9a50-468802aa4fc1-model-cache\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:25.066377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.066344    5085 generic.go:356] "Generic (PLEG): container finished" podID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerID="a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016" exitCode=0
Oct 30 20:42:25.066377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.066384    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"
Oct 30 20:42:25.066612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.066420    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" event={"ID":"5dfd1e13-ef55-4072-9a50-468802aa4fc1","Type":"ContainerDied","Data":"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"}
Oct 30 20:42:25.066612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.066463    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx" event={"ID":"5dfd1e13-ef55-4072-9a50-468802aa4fc1","Type":"ContainerDied","Data":"600861b15327c0e24eaabc10909940def6fdcf39f164522c6e72224b6e87520a"}
Oct 30 20:42:25.066612 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.066483    5085 scope.go:122] "RemoveContainer" containerID="a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"
Oct 30 20:42:25.068205 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.068182    5085 generic.go:356] "Generic (PLEG): container finished" podID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerID="24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9" exitCode=0
Oct 30 20:42:25.068310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.068222    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"
Oct 30 20:42:25.068310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.068254    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" event={"ID":"69ce92c4-2611-41d2-91da-1f76052c76d6","Type":"ContainerDied","Data":"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"}
Oct 30 20:42:25.068310 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.068279    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" event={"ID":"69ce92c4-2611-41d2-91da-1f76052c76d6","Type":"ContainerDied","Data":"9c4af9dcb87c750e82d514d359557af61ec8e030002326d112c8eca05074b293"}
Oct 30 20:42:25.076634 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.076615    5085 scope.go:122] "RemoveContainer" containerID="a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"
Oct 30 20:42:25.076985 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:25.076961    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016\": container with ID starting with a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016 not found: ID does not exist" containerID="a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"
Oct 30 20:42:25.077118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.076992    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016"} err="failed to get container status \"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016\": rpc error: code = NotFound desc = could not find container \"a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016\": container with ID starting with a5cd15c1193d7909b7ae03898b80e487e273281c1c74460e5556170128294016 not found: ID does not exist"
Oct 30 20:42:25.077118 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.077009    5085 scope.go:122] "RemoveContainer" containerID="24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"
Oct 30 20:42:25.087308 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.086695    5085 scope.go:122] "RemoveContainer" containerID="24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"
Oct 30 20:42:25.087308 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:25.087085    5085 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9\": container with ID starting with 24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9 not found: ID does not exist" containerID="24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"
Oct 30 20:42:25.087308 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.087108    5085 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9"} err="failed to get container status \"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9\": rpc error: code = NotFound desc = could not find container \"24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9\": container with ID starting with 24a4b9f221477463523e60a872f93fa11f0dadff96906c90a66a24dff1f3a5d9 not found: ID does not exist"
Oct 30 20:42:25.189591 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.189554    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:25.235608 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.235574    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\" (UniqueName: \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\") pod \"644e9ab7-363f-420d-9d65-f66b5ab8164f\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") "
Oct 30 20:42:25.235780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.235668    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\") pod \"644e9ab7-363f-420d-9d65-f66b5ab8164f\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") "
Oct 30 20:42:25.235780 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.235735    5085 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\") pod \"644e9ab7-363f-420d-9d65-f66b5ab8164f\" (UID: \"644e9ab7-363f-420d-9d65-f66b5ab8164f\") "
Oct 30 20:42:25.236113 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.236086    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 20:42:25.236205 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.236121    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca" pod "644e9ab7-363f-420d-9d65-f66b5ab8164f" (UID: "644e9ab7-363f-420d-9d65-f66b5ab8164f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:42:25.237719 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.237700    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9" pod "644e9ab7-363f-420d-9d65-f66b5ab8164f" (UID: "644e9ab7-363f-420d-9d65-f66b5ab8164f"). InnerVolumeSpecName "kube-api-access-7n8g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:42:25.237842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.237812    5085 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca" pod "644e9ab7-363f-420d-9d65-f66b5ab8164f" (UID: "644e9ab7-363f-420d-9d65-f66b5ab8164f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:42:25.240810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.240786    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9"]
Oct 30 20:42:25.337270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.337190    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-xks-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:25.337270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.337218    5085 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644e9ab7-363f-420d-9d65-f66b5ab8164f-maas-api-openshift-ca\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:25.337270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.337230    5085 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n8g9\" (UniqueName: \"kubernetes.io/projected/644e9ab7-363f-420d-9d65-f66b5ab8164f-kube-api-access-7n8g9\") on node \"ip-10-0-134-200.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 20:42:25.353163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.353128    5085 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 20:42:25.358211 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.358186    5085 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b0a183/distinct-2-w2-b0a183-kserve-79fcf5b6b5-sw9sx"]
Oct 30 20:42:25.523357 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:25.523312    5085 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b0a183/simulator-w2-b0a183-kserve-677b4cc58-gxkb9" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:42:26.075630 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:26.075603    5085 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5"
Oct 30 20:42:26.075833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:26.075603    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-890606b6-kzmx5" event={"ID":"644e9ab7-363f-420d-9d65-f66b5ab8164f","Type":"ContainerDied","Data":"33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18"}
Oct 30 20:42:26.075833 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:26.075744    5085 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33190f5926e567f4ca49b1121b8204d3b4bf184015f3903be966867ae22e9e18"
Oct 30 20:42:26.630659 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:26.630626    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" path="/var/lib/kubelet/pods/5dfd1e13-ef55-4072-9a50-468802aa4fc1/volumes"
Oct 30 20:42:26.631130 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:26.631114    5085 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" path="/var/lib/kubelet/pods/69ce92c4-2611-41d2-91da-1f76052c76d6/volumes"
Oct 30 20:42:37.487001 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:37.486963    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:42:49.785161 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:49.785124    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:42:50.120265 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:50.120185    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5r6fq_bafa56a2-5852-481e-bed8-94df375b14b6/manager/0.log"
Oct 30 20:42:50.358382 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:50.358352    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gz5b9_eeb9cbac-453c-440e-9608-2b80fb7424d4/manager/0.log"
Oct 30 20:42:51.619100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:51.619065    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-mfq8q"]
Oct 30 20:42:51.938815 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:42:51.938779    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:42:52.099390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:52.099365    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mfq8q_46f9d21e-6c67-44c5-a817-32f250962178/limitador/0.log"
Oct 30 20:42:52.349986 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:52.349909    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jt8mk_497a76db-4f65-47d2-9d8d-857e278887dc/manager/0.log"
Oct 30 20:42:53.179150 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:53.179117    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:42:53.427939 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:53.427902    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:42:54.564355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:54.564329    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:42:54.788724 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:54.788696    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:42:55.810752 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:42:55.810720    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:43:06.141240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:06.141208    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:06.497806 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:06.497778    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:07.634611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:07.634585    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:07.858377 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:07.858345    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:09.635636 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:09.635608    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-890606b6-kzmx5_644e9ab7-363f-420d-9d65-f66b5ab8164f/revoke-keys/0.log"
Oct 30 20:43:10.294052 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:10.294027    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-gldql_976d229d-31d9-40b9-907b-0b459a957b82/postgres/0.log"
Oct 30 20:43:11.766684 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:11.766657    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:43:12.078432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:12.078355    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5r6fq_bafa56a2-5852-481e-bed8-94df375b14b6/manager/0.log"
Oct 30 20:43:12.305552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:12.305520    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gz5b9_eeb9cbac-453c-440e-9608-2b80fb7424d4/manager/0.log"
Oct 30 20:43:13.632869 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:13.632841    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mfq8q_46f9d21e-6c67-44c5-a817-32f250962178/limitador/0.log"
Oct 30 20:43:13.852688 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:13.852657    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jt8mk_497a76db-4f65-47d2-9d8d-857e278887dc/manager/0.log"
Oct 30 20:43:14.655767 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:14.655738    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt_df5d6579-23f1-4728-8af6-8bdf452d8e96/istio-proxy/0.log"
Oct 30 20:43:15.562933 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:15.562901    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/1.log"
Oct 30 20:43:15.798349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:15.798316    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg_dbd27053-72dc-4215-885b-32f291f98853/istio-proxy/0.log"
Oct 30 20:43:16.013403 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:16.013376    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-j5xnn_23ec560d-02a4-4c34-80c4-4a7e58bd5874/payload-pre-processing/0.log"
Oct 30 20:43:16.233625 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:16.233600    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:43:16.466165 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:16.466129    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8598d5bb4c-4xl9z_d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/router/0.log"
Oct 30 20:43:16.946079 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:16.946049    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/0.log"
Oct 30 20:43:17.519487 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:17.519458    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-w5ns9_0f6cb9b2-60f1-4817-813c-24c9850d6075/main/0.log"
Oct 30 20:43:17.754198 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:17.754169    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-b5vhn_638c345b-06f3-400f-8c71-f6bfe6dd98bf/main/0.log"
Oct 30 20:43:18.174656 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:18.174622    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qqqrm_41d83157-4461-44cd-a4b3-804bc7089c77/main/0.log"
Oct 30 20:43:18.861393 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:18.861368    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8jfsh_17151fda-8d13-423b-9fba-b177eb796463/main/0.log"
Oct 30 20:43:19.077810 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:19.077779    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ws6th_00a40e56-bd58-4960-98ab-e6f7be89e798/main/0.log"
Oct 30 20:43:21.939092 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:43:21.939060    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:43:29.189971 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.189886    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j795s/must-gather-lfz5d"]
Oct 30 20:43:29.190391 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190376    5085 cpu_manager.go:414] "containerMap: removing container" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main"
Oct 30 20:43:29.190455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190392    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main"
Oct 30 20:43:29.190455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190404    5085 cpu_manager.go:414] "containerMap: removing container" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main"
Oct 30 20:43:29.190455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190409    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main"
Oct 30 20:43:29.190455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190416    5085 cpu_manager.go:414] "containerMap: removing container" podUID="644e9ab7-363f-420d-9d65-f66b5ab8164f" containerName="revoke-keys"
Oct 30 20:43:29.190455 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190421    5085 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644e9ab7-363f-420d-9d65-f66b5ab8164f" containerName="revoke-keys"
Oct 30 20:43:29.190678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190499    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfd1e13-ef55-4072-9a50-468802aa4fc1" containerName="main"
Oct 30 20:43:29.190678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190509    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ce92c4-2611-41d2-91da-1f76052c76d6" containerName="main"
Oct 30 20:43:29.190678 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.190514    5085 memory_manager.go:365] "RemoveStaleState removing state" podUID="644e9ab7-363f-420d-9d65-f66b5ab8164f" containerName="revoke-keys"
Oct 30 20:43:29.193819 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.193799    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.244721 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.244685    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqtms\" (UniqueName: \"kubernetes.io/projected/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-kube-api-access-gqtms\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.244903 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.244826    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-must-gather-output\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.300134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.300104    5085 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j795s\"/\"default-dockercfg-bsc74\""
Oct 30 20:43:29.300314 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.300155    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j795s\"/\"openshift-service-ca.crt\""
Oct 30 20:43:29.300314 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.300191    5085 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j795s\"/\"kube-root-ca.crt\""
Oct 30 20:43:29.302692 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.302667    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/must-gather-lfz5d"]
Oct 30 20:43:29.346060 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.346028    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-must-gather-output\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.346275 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.346116    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqtms\" (UniqueName: \"kubernetes.io/projected/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-kube-api-access-gqtms\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.346390 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.346370    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-must-gather-output\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.408127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.408092    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqtms\" (UniqueName: \"kubernetes.io/projected/11f852c5-25bf-44d3-a8ab-b107e5ba7c68-kube-api-access-gqtms\") pod \"must-gather-lfz5d\" (UID: \"11f852c5-25bf-44d3-a8ab-b107e5ba7c68\") " pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.507202 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.507116    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/must-gather-lfz5d"
Oct 30 20:43:29.882659 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:43:29.882622    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f852c5_25bf_44d3_a8ab_b107e5ba7c68.slice/crio-723d6841542a4294367e486c732612acc6410e152161c82ef8b7303080962254 WatchSource:0}: Error finding container 723d6841542a4294367e486c732612acc6410e152161c82ef8b7303080962254: Status 404 returned error can't find the container with id 723d6841542a4294367e486c732612acc6410e152161c82ef8b7303080962254
Oct 30 20:43:29.885240 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.885203    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/must-gather-lfz5d"]
Oct 30 20:43:29.888982 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:29.888961    5085 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:43:30.467932 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:30.467902    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/must-gather-lfz5d" event={"ID":"11f852c5-25bf-44d3-a8ab-b107e5ba7c68","Type":"ContainerStarted","Data":"723d6841542a4294367e486c732612acc6410e152161c82ef8b7303080962254"}
Oct 30 20:43:32.481990 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:32.481958    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/must-gather-lfz5d" event={"ID":"11f852c5-25bf-44d3-a8ab-b107e5ba7c68","Type":"ContainerStarted","Data":"30edecf4d1700cd68d5ec2a8144a32011dbb002b1f9ad79b349775e1951ab3eb"}
Oct 30 20:43:32.482356 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:32.481999    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/must-gather-lfz5d" event={"ID":"11f852c5-25bf-44d3-a8ab-b107e5ba7c68","Type":"ContainerStarted","Data":"26b1c785c03bf573a687c8f3e1219ec58a4b57aa67fab2aa59b3dbb8d5cb2259"}
Oct 30 20:43:32.601492 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:32.601445    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j795s/must-gather-lfz5d" podStartSLOduration=1.9365661539999999 podStartE2EDuration="3.601431183s" totalImagesPullingTime="1.664865029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:32.598960742 +0000 UTC m=+4722.103364492" watchObservedRunningTime="2026-10-30 20:43:32.601431183 +0000 UTC m=+4722.105834944"
Oct 30 20:43:39.718516 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:39.718479    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vg85t_4d9e730e-03fe-43f4-983e-00a9e6975b6e/global-pull-secret-syncer/0.log"
Oct 30 20:43:40.938100 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:40.938070    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-ldqhw_8c679173-6012-49d3-9a8b-f1b846552a35/connection-checker/0.log"
Oct 30 20:43:41.364635 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:41.362867    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rtrx2_45a28670-76ac-4e67-be4b-72e0a64e6766/connection-checker/0.log"
Oct 30 20:43:41.426753 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:41.426714    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:41.626005 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:41.625919    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:41.829609 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:41.829552    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xx99q_11477705-d48e-4ba3-a15a-8aba013e83cc/connection-checker/0.log"
Oct 30 20:43:42.720586 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:42.716994    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:43.175122 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:43.175079    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhxdt_d690fb63-2b99-4a20-8a76-4650ce687a0f/konnectivity-agent/0.log"
Oct 30 20:43:43.872432 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:43.872401    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:44.259857 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:44.259822    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:44.393049 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:44.393014    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-200.us-west-2.compute.internal_01a2322c0b6f8492dadef7e7f75813b1/haproxy/0.log"
Oct 30 20:43:44.449284 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:44.449254    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:45.548121 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:45.548089    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:46.619158 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:46.619127    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:47.089647 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:47.089616    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:47.283406 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:47.283356    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:48.387587 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:48.386980    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:49.025875 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.025830    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:43:49.358138 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.358061    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5r6fq_bafa56a2-5852-481e-bed8-94df375b14b6/manager/0.log"
Oct 30 20:43:49.480370 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.480334    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:49.640637 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.640545    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:49.673225 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.673201    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:49.781337 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.781308    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gz5b9_eeb9cbac-453c-440e-9608-2b80fb7424d4/manager/0.log"
Oct 30 20:43:49.900127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.900047    5085 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"]
Oct 30 20:43:49.908920 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.908891    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:49.921279 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:43:49.921246    5085 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-pod416cb084_106c_4469_9778_09eff8cb1f7d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 20:43:49.965410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.965369    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb94b\" (UniqueName: \"kubernetes.io/projected/416cb084-106c-4469-9778-09eff8cb1f7d-kube-api-access-hb94b\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:49.965410 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.965416    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-sys\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:49.965667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.965462    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-lib-modules\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:49.965667 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.965635    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-proc\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:49.965745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:49.965666    5085 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-podres\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.003324 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.003290    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"]
Oct 30 20:43:50.066476 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066436    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-proc\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066482    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-podres\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066581    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-proc\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066595    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb94b\" (UniqueName: \"kubernetes.io/projected/416cb084-106c-4469-9778-09eff8cb1f7d-kube-api-access-hb94b\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066644    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-podres\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066645    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-sys\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066689    5085 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-lib-modules\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.066711 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066691    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-sys\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.067004 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.066751    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/416cb084-106c-4469-9778-09eff8cb1f7d-lib-modules\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.129681 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.129653    5085 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb94b\" (UniqueName: \"kubernetes.io/projected/416cb084-106c-4469-9778-09eff8cb1f7d-kube-api-access-hb94b\") pod \"perf-node-gather-daemonset-cvx5r\" (UID: \"416cb084-106c-4469-9778-09eff8cb1f7d\") " pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.231355 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.231321    5085 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:50.612001 ip-10-0-134-200 kubenswrapper[5085]: W1030 20:43:50.611966    5085 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod416cb084_106c_4469_9778_09eff8cb1f7d.slice/crio-3f2a8c1724e8e34d1a901fd67a9d9312768f11ad931253c3e26401af6304a06a WatchSource:0}: Error finding container 3f2a8c1724e8e34d1a901fd67a9d9312768f11ad931253c3e26401af6304a06a: Status 404 returned error can't find the container with id 3f2a8c1724e8e34d1a901fd67a9d9312768f11ad931253c3e26401af6304a06a
Oct 30 20:43:50.614909 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.614884    5085 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"]
Oct 30 20:43:50.637173 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.637132    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r" event={"ID":"416cb084-106c-4469-9778-09eff8cb1f7d","Type":"ContainerStarted","Data":"3f2a8c1724e8e34d1a901fd67a9d9312768f11ad931253c3e26401af6304a06a"}
Oct 30 20:43:50.883270 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:50.883177    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:51.647552 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:51.647452    5085 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r" event={"ID":"416cb084-106c-4469-9778-09eff8cb1f7d","Type":"ContainerStarted","Data":"6cfa8a75574a6ce86cdfcf6db766b7531db6361bd5b1e81af64c3a9f7d7c1bb1"}
Oct 30 20:43:51.648611 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:51.648584    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:51.707800 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:51.707738    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mfq8q_46f9d21e-6c67-44c5-a817-32f250962178/limitador/0.log"
Oct 30 20:43:51.768906 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:51.768837    5085 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r" podStartSLOduration=2.768820183 podStartE2EDuration="2.768820183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:51.767525114 +0000 UTC m=+4741.271928863" watchObservedRunningTime="2026-10-30 20:43:51.768820183 +0000 UTC m=+4741.273223930"
Oct 30 20:43:51.857858 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:51.857823    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:51.938310 ip-10-0-134-200 kubenswrapper[5085]: E1030 20:43:51.938267    5085 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qbzm8" containerName="extractor"
Oct 30 20:43:52.186236 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:52.186201    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jt8mk_497a76db-4f65-47d2-9d8d-857e278887dc/manager/0.log"
Oct 30 20:43:52.754524 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:52.754494    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:52.950317 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:52.950283    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:54.050518 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:54.050491    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:55.017777 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:55.017748    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:55.024376 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:55.024354    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76fc486898-xkrnb_0a4684b7-2357-4383-ac16-c5a6b3be17ee/manager/0.log"
Oct 30 20:43:55.218050 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:55.218019    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-xwf7b_f7e56e43-f4e7-4719-9c01-2e8d228a773f/manager/0.log"
Oct 30 20:43:56.348325 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:56.348301    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-x4rdp_f7b387ef-f51d-416a-a75e-ce2560713559/server/0.log"
Oct 30 20:43:57.275729 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:57.275693    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pc2dt_e2f10a7d-5fae-4943-9d9e-c13705842b79/manager/0.log"
Oct 30 20:43:58.134171 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:58.134140    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-n9x5b_51992eb3-1df9-4e7f-9541-a9e5c8aab9f7/cluster-monitoring-operator/0.log"
Oct 30 20:43:58.671194 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:58.671164    5085 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j795s/perf-node-gather-daemonset-cvx5r"
Oct 30 20:43:58.771745 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:43:58.771715    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt_df5d6579-23f1-4728-8af6-8bdf452d8e96/istio-proxy/0.log"
Oct 30 20:44:00.118135 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.118107    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/0.log"
Oct 30 20:44:00.120243 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.120224    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/1.log"
Oct 30 20:44:00.229182 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.229150    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg_dbd27053-72dc-4215-885b-32f291f98853/istio-proxy/0.log"
Oct 30 20:44:00.445622 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.437371    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-j5xnn_23ec560d-02a4-4c34-80c4-4a7e58bd5874/payload-pre-processing/0.log"
Oct 30 20:44:00.645134 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.645102    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:44:00.864349 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.864278    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8598d5bb4c-4xl9z_d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/router/0.log"
Oct 30 20:44:00.976163 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:00.976130    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt_df5d6579-23f1-4728-8af6-8bdf452d8e96/istio-proxy/0.log"
Oct 30 20:44:02.380232 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:02.380204    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/0.log"
Oct 30 20:44:02.382469 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:02.382435    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-694db55c87-mcbsf_f7eff08b-642e-4125-90e1-88ea8adf5385/istio-proxy/1.log"
Oct 30 20:44:02.495466 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:02.495439    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856cdd999f-k4nlg_dbd27053-72dc-4215-885b-32f291f98853/istio-proxy/0.log"
Oct 30 20:44:02.705842 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:02.705811    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-j5xnn_23ec560d-02a4-4c34-80c4-4a7e58bd5874/payload-pre-processing/0.log"
Oct 30 20:44:02.917335 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:02.917307    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-t5gqn_049e0e86-319c-4386-99ee-f13821b9c079/payload-processing/0.log"
Oct 30 20:44:03.135127 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:03.135054    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8598d5bb4c-4xl9z_d9b9a83a-3e00-46aa-9cdd-4508a3f917eb/router/0.log"
Oct 30 20:44:03.439146 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:03.439115    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkrcl_44fc5a64-c15d-4da6-b787-7a3fbe86f12e/node-exporter/0.log"
Oct 30 20:44:03.864804 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:03.864726    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkrcl_44fc5a64-c15d-4da6-b787-7a3fbe86f12e/kube-rbac-proxy/0.log"
Oct 30 20:44:04.289741 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:04.289716    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rkrcl_44fc5a64-c15d-4da6-b787-7a3fbe86f12e/init-textfile/0.log"
Oct 30 20:44:06.205957 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:06.205919    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:44:06.316417 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:06.316392    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5r6fq_bafa56a2-5852-481e-bed8-94df375b14b6/manager/0.log"
Oct 30 20:44:06.524435 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:06.524361    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gz5b9_eeb9cbac-453c-440e-9608-2b80fb7424d4/manager/0.log"
Oct 30 20:44:07.640533 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:07.640501    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mfq8q_46f9d21e-6c67-44c5-a817-32f250962178/limitador/0.log"
Oct 30 20:44:07.910803 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:07.910723    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jt8mk_497a76db-4f65-47d2-9d8d-857e278887dc/manager/0.log"
Oct 30 20:44:08.471033 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:08.470998    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fcd7f7c4-p4ccs_559f8fac-bb0f-47c6-8bf7-12b98a1d50fb/authorino/0.log"
Oct 30 20:44:08.587138 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:08.587107    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5r6fq_bafa56a2-5852-481e-bed8-94df375b14b6/manager/0.log"
Oct 30 20:44:08.793439 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:08.793348    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-gz5b9_eeb9cbac-453c-440e-9608-2b80fb7424d4/manager/0.log"
Oct 30 20:44:09.935175 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:09.935145    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-mfq8q_46f9d21e-6c67-44c5-a817-32f250962178/limitador/0.log"
Oct 30 20:44:10.184941 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:10.184766    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jt8mk_497a76db-4f65-47d2-9d8d-857e278887dc/manager/0.log"
Oct 30 20:44:13.817689 ip-10-0-134-200 kubenswrapper[5085]: I1030 20:44:13.817660    5085 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-c4c65fbf-cnvwt_df5d6579-23f1-4728-8af6-8bdf452d8e96/istio-proxy/0.log"